Note: This is test shard 3 of 4. [==========] Running 22 tests from 4 test cases. [----------] Global test environment set-up. [----------] 18 tests from ToolTest [ RUN ] ToolTest.TestModeHelp [ OK ] ToolTest.TestModeHelp (4485 ms) [ RUN ] ToolTest.TestFsCheckLiveServer 2020-05-21T22:28:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-05-21T22:28:29Z Disabled control of system clock WARNING: Logging before InitGoogleLogging() is written to STDERR I0521 22:28:29.195937 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --ipki_ca_key_size=768 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:39393 --webserver_interface=127.0.89.254 --webserver_port=0 --builtin_ntp_servers=127.0.89.212:37691 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:28:29.340930 393 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:28:29.341473 393 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:28:29.341938 393 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:28:29.354707 393 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=768 W0521 22:28:29.354915 393 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:28:29.355151 393 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0521 22:28:29.355388 393 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0521 22:28:29.374708 393 master_runner.cc:93] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:37691 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/wal --ipki_ca_key_size=768 --ipki_server_key_size=768 --openssl_security_level_override=0 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:39393 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.89.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:28:29.375658 393 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:28:29.377930 393 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:28:29.393337 393 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:28:29.397045 393 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:28:29.398366 393 hybrid_clock.cc:590] HybridClock initialized: now 1590100109398324 us; error 53 us; skew 500 ppm I0521 22:28:29.400761 393 fs_manager.cc:259] Metadata directory not provided I0521 22:28:29.400926 393 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:28:29.401273 393 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:28:29.404438 393 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/data/instance: uuid: "32a2200abe0a45aea3f002b254726a20" format_stamp: "Formatted at 2020-05-21 22:28:29 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:29.405234 393 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/wal/instance: uuid: "32a2200abe0a45aea3f002b254726a20" format_stamp: "Formatted at 2020-05-21 22:28:29 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:29.425616 393 fs_manager.cc:518] Time spent creating directory manager: real 0.020s user 0.017s sys 0.003s I0521 22:28:29.433712 393 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:28:29.433864 393 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/wal uuid: "32a2200abe0a45aea3f002b254726a20" format_stamp: "Formatted at 2020-05-21 22:28:29 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:29.434159 393 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:28:30.195894 393 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.728s user 0.000s sys 0.001s W0521 22:28:30.196138 393 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.728s user 0.000s sys 0.002s I0521 22:28:30.213683 393 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:28:30.309816 393 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.254:39393 I0521 22:28:30.313423 393 webserver.cc:381] Webserver started at http://127.0.89.254:37919/ using document root and password file I0521 22:28:30.314390 393 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/master-0/data/info.pb I0521 22:28:30.319900 455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:28:30.322556 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master as pid 393 I0521 22:28:30.325069 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --rpc_bind_addresses=127.0.89.193:0 --local_ip_for_outbound_sockets=127.0.89.193 --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:39393 --builtin_ntp_servers=127.0.89.212:37691 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} I0521 22:28:30.346753 455 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20: Bootstrap starting. I0521 22:28:30.353724 455 tablet_bootstrap.cc:596] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20: No blocks or log segments found. Creating new log. I0521 22:28:30.355746 455 log.cc:819] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20: Log is configured to *not* fsync() on all Append() calls I0521 22:28:30.363050 455 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20: No bootstrap required, opened a new log I0521 22:28:30.390753 455 raft_consensus.cc:342] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a2200abe0a45aea3f002b254726a20" member_type: VOTER } I0521 22:28:30.391347 455 raft_consensus.cc:368] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:28:30.391590 455 raft_consensus.cc:722] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a2200abe0a45aea3f002b254726a20, State: Initialized, Role: FOLLOWER I0521 22:28:30.392400 455 consensus_queue.cc:250] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a2200abe0a45aea3f002b254726a20" member_type: VOTER } I0521 22:28:30.392951 455 raft_consensus.cc:382] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:28:30.393255 455 raft_consensus.cc:478] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:28:30.393599 455 raft_consensus.cc:2975] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:28:30.399901 455 raft_consensus.cc:500] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a2200abe0a45aea3f002b254726a20" member_type: VOTER } I0521 22:28:30.400660 455 leader_election.cc:302] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32a2200abe0a45aea3f002b254726a20; no voters: I0521 22:28:30.404410 455 leader_election.cc:288] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:28:30.404727 458 raft_consensus.cc:2721] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:28:30.544296 458 raft_consensus.cc:683] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [term 1 LEADER]: Becoming Leader. State: Replica: 32a2200abe0a45aea3f002b254726a20, State: Running, Role: LEADER I0521 22:28:30.544865 458 consensus_queue.cc:227] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a2200abe0a45aea3f002b254726a20" member_type: VOTER } I0521 22:28:30.551648 460 sys_catalog.cc:357] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32a2200abe0a45aea3f002b254726a20. Latest consensus state: current_term: 1 leader_uuid: "32a2200abe0a45aea3f002b254726a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a2200abe0a45aea3f002b254726a20" member_type: VOTER } } I0521 22:28:30.551649 459 sys_catalog.cc:357] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32a2200abe0a45aea3f002b254726a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32a2200abe0a45aea3f002b254726a20" member_type: VOTER } } I0521 22:28:30.552378 459 sys_catalog.cc:360] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [sys.catalog]: This master's current role is: LEADER I0521 22:28:30.552384 460 sys_catalog.cc:360] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [sys.catalog]: This master's current role is: LEADER I0521 22:28:30.553679 455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20 [sys.catalog]: configured and running, proceeding with master startup. WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:28:30.562904 456 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:28:30.563326 456 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:28:30.563841 456 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:28:30.571599 464 catalog_manager.cc:1201] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0521 22:28:30.571858 464 catalog_manager.cc:667] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0521 22:28:30.572229 462 catalog_manager.cc:1131] Loading table and tablet metadata into memory... W0521 22:28:30.578052 456 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:28:30.580008 456 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.89.193 I0521 22:28:30.580430 462 catalog_manager.cc:1142] Initializing Kudu internal certificate authority... I0521 22:28:30.600910 456 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:37691 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/wal --ipki_server_key_size=768 --openssl_security_level_override=0 --rpc_bind_addresses=127.0.89.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:39393 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.89.193 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:28:30.601867 456 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:28:30.603255 456 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:28:30.619658 456 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:28:30.622040 462 catalog_manager.cc:1026] Generated new certificate authority record I0521 22:28:30.625469 462 catalog_manager.cc:1151] Loading token signing keys... I0521 22:28:30.636377 456 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:28:30.643959 456 hybrid_clock.cc:590] HybridClock initialized: now 1590100110643915 us; error 46 us; skew 500 ppm I0521 22:28:30.648080 462 catalog_manager.cc:4538] T 00000000000000000000000000000000 P 32a2200abe0a45aea3f002b254726a20: Generated new TSK 0 I0521 22:28:30.649128 462 catalog_manager.cc:1161] Initializing in-progress tserver states... I0521 22:28:30.656841 456 fs_manager.cc:259] Metadata directory not provided I0521 22:28:30.656997 456 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:28:30.657353 456 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:28:30.660524 456 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/data/instance: uuid: "7b34313caf3b4b2faf1b75a5f512cbf9" format_stamp: "Formatted at 2020-05-21 22:28:30 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:30.661322 456 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/wal/instance: uuid: "7b34313caf3b4b2faf1b75a5f512cbf9" format_stamp: "Formatted at 2020-05-21 22:28:30 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:30.681970 456 fs_manager.cc:518] Time spent creating directory manager: real 0.020s user 0.013s sys 0.008s I0521 22:28:30.690490 456 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:28:30.690642 456 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/wal uuid: "7b34313caf3b4b2faf1b75a5f512cbf9" format_stamp: "Formatted at 2020-05-21 22:28:30 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:30.690958 456 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:28:31.695451 456 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.964s user 0.001s sys 0.001s W0521 22:28:31.695763 456 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.964s user 0.002s sys 0.002s I0521 22:28:31.725437 456 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:28:31.735997 456 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:28:31.736277 456 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:28:32.078346 456 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.193:45755 I0521 22:28:32.092487 456 webserver.cc:381] Webserver started at http://127.0.89.193:43397/ using document root and password file I0521 22:28:32.093490 456 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestFsCheckLiveServer.1590100104622499-359/minicluster-data/ts-0/data/info.pb I0521 22:28:32.097227 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver as pid 456 I0521 22:28:32.140060 588 heartbeater.cc:325] Connected to a master server at 127.0.89.254:39393 I0521 22:28:32.140492 588 heartbeater.cc:416] Registering TS with master... I0521 22:28:32.141453 588 heartbeater.cc:465] Master 127.0.89.254:39393 requested a full tablet report, sending... I0521 22:28:32.144914 422 ts_manager.cc:191] Registered new tserver with Master: 7b34313caf3b4b2faf1b75a5f512cbf9 (127.0.89.193:45755) I0521 22:28:32.146348 359 external_mini_cluster.cc:699] 1 TS(s) registered with all masters I0521 22:28:32.147676 422 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.89.193:41921 I0521 22:28:33.151887 588 heartbeater.cc:457] Master 127.0.89.254:39393 was elected leader, sending a full tablet report... I0521 22:28:34.698156 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver with pid 456 I0521 22:28:34.711514 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master with pid 393 2020-05-21T22:28:34Z chronyd exiting [ OK ] ToolTest.TestFsCheckLiveServer (5598 ms) [ RUN ] ToolTest.TestPbcTools I0521 22:28:34.759965 359 fs_manager.cc:259] Metadata directory not provided I0521 22:28:34.760150 359 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:28:34.763720 359 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPbcTools.1590100104622499-359/test/instance: uuid: "caf2f36dc97444b5a7517a267067a2e8" format_stamp: "Formatted at 2020-05-21 22:28:34 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:34.785419 359 fs_manager.cc:518] Time spent creating directory manager: real 0.021s user 0.017s sys 0.003s I0521 22:28:34.793176 359 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPbcTools.1590100104622499-359/test/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0521 22:28:34.793606 359 fs_manager.cc:433] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0521 22:28:34.793743 359 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPbcTools.1590100104622499-359/test uuid: "caf2f36dc97444b5a7517a267067a2e8" format_stamp: "Formatted at 2020-05-21 22:28:34 on dist-test-slave-dist-test-slave-jp1g" [ OK ] ToolTest.TestPbcTools (20167 ms) [ RUN ] ToolTest.TestLocalReplicaDumpDataDirs I0521 22:28:54.923450 359 fs_manager.cc:259] Metadata directory not provided I0521 22:28:54.923624 359 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:28:54.924800 359 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDumpDataDirs.1590100104622499-359/data0/instance: uuid: "a65b16efa0c6408999ab61bb1e52ba3a" format_stamp: "Formatted at 2020-05-21 22:28:54 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:54.925606 359 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDumpDataDirs.1590100104622499-359/data1/instance: uuid: "a65b16efa0c6408999ab61bb1e52ba3a" format_stamp: "Formatted at 2020-05-21 22:28:54 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:54.926321 359 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDumpDataDirs.1590100104622499-359/data2/instance: uuid: "a65b16efa0c6408999ab61bb1e52ba3a" format_stamp: "Formatted at 2020-05-21 22:28:54 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:54.927027 359 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDumpDataDirs.1590100104622499-359/wal/instance: uuid: "a65b16efa0c6408999ab61bb1e52ba3a" format_stamp: "Formatted at 2020-05-21 22:28:54 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:54.953923 359 fs_manager.cc:518] Time spent creating directory manager: real 0.026s user 0.026s sys 0.003s I0521 22:28:54.962766 359 fs_report.cc:352] FS layout report -------------------- wal directory: metadata directory: 3 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDumpDataDirs.1590100104622499-359/data0/data, /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDumpDataDirs.1590100104622499-359/data1/data, /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDumpDataDirs.1590100104622499-359/data2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0521 22:28:54.963158 359 fs_manager.cc:433] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0521 22:28:54.963306 359 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDumpDataDirs.1590100104622499-359/data0,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDumpDataDirs.1590100104622499-359/data1,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDumpDataDirs.1590100104622499-359/data2,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDumpDataDirs.1590100104622499-359/wal uuid: "a65b16efa0c6408999ab61bb1e52ba3a" format_stamp: "Formatted at 2020-05-21 22:28:54 on dist-test-slave-dist-test-slave-jp1g" [ OK ] ToolTest.TestLocalReplicaDumpDataDirs (2066 ms) [ RUN ] ToolTest.TestLoadgenDefaultParameters 2020-05-21T22:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-05-21T22:28:56Z Disabled control of system clock I0521 22:28:57.001402 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --ipki_ca_key_size=768 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:37037 --webserver_interface=127.0.89.254 --webserver_port=0 --builtin_ntp_servers=127.0.89.212:35875 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:28:57.147614 688 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:28:57.148186 688 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:28:57.148706 688 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:28:57.161176 688 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=768 W0521 22:28:57.161353 688 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:28:57.161554 688 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0521 22:28:57.161746 688 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0521 22:28:57.180732 688 master_runner.cc:93] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:35875 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/wal --ipki_ca_key_size=768 --ipki_server_key_size=768 --openssl_security_level_override=0 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:37037 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.89.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:28:57.181641 688 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:28:57.183110 688 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:28:57.198029 688 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:28:57.201161 688 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:28:57.202450 688 hybrid_clock.cc:590] HybridClock initialized: now 1590100137202419 us; error 39 us; skew 500 ppm I0521 22:28:57.204500 688 fs_manager.cc:259] Metadata directory not provided I0521 22:28:57.204661 688 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:28:57.205004 688 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:28:57.208087 688 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/data/instance: uuid: "528bb2d9313a499aaa0182c4e5d3606d" format_stamp: "Formatted at 2020-05-21 22:28:57 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:57.210016 688 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/wal/instance: uuid: "528bb2d9313a499aaa0182c4e5d3606d" format_stamp: "Formatted at 2020-05-21 22:28:57 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:57.229848 688 fs_manager.cc:518] Time spent creating directory manager: real 0.019s user 0.017s sys 0.002s I0521 22:28:57.237972 688 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:28:57.238111 688 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/wal uuid: "528bb2d9313a499aaa0182c4e5d3606d" format_stamp: "Formatted at 2020-05-21 22:28:57 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:57.238394 688 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:28:58.015271 688 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.750s user 0.362s sys 0.386s W0521 22:28:58.015547 688 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.750s user 0.362s sys 0.387s I0521 22:28:58.038668 688 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:28:58.104100 688 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.254:37037 I0521 22:28:58.108052 688 webserver.cc:381] Webserver started at http://127.0.89.254:45689/ using document root and password file I0521 22:28:58.109364 688 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/master-0/data/info.pb I0521 22:28:58.113785 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master as pid 688 I0521 22:28:58.114190 750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:28:58.116154 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --rpc_bind_addresses=127.0.89.193:0 --local_ip_for_outbound_sockets=127.0.89.193 --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:37037 --builtin_ntp_servers=127.0.89.212:35875 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} I0521 22:28:58.138752 750 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: Bootstrap starting. I0521 22:28:58.145284 750 tablet_bootstrap.cc:596] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: No blocks or log segments found. Creating new log. I0521 22:28:58.147457 750 log.cc:819] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: Log is configured to *not* fsync() on all Append() calls I0521 22:28:58.155861 750 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: No bootstrap required, opened a new log I0521 22:28:58.181474 750 raft_consensus.cc:342] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528bb2d9313a499aaa0182c4e5d3606d" member_type: VOTER } I0521 22:28:58.182076 750 raft_consensus.cc:368] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:28:58.182348 750 raft_consensus.cc:722] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528bb2d9313a499aaa0182c4e5d3606d, State: Initialized, Role: FOLLOWER I0521 22:28:58.183157 750 consensus_queue.cc:250] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528bb2d9313a499aaa0182c4e5d3606d" member_type: VOTER } I0521 22:28:58.183732 750 raft_consensus.cc:382] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:28:58.183969 750 raft_consensus.cc:478] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:28:58.184263 750 raft_consensus.cc:2975] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [term 0 FOLLOWER]: Advancing to term 1 WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:28:58.244346 751 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:28:58.244768 751 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:28:58.245285 751 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:28:58.255738 751 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:28:58.256691 751 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.89.193 I0521 22:28:58.272897 751 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:35875 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/wal --ipki_server_key_size=768 --openssl_security_level_override=0 --rpc_bind_addresses=127.0.89.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:37037 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.89.193 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:28:58.273829 751 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:28:58.275252 751 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:28:58.290740 751 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:28:58.293771 751 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:28:58.295135 751 hybrid_clock.cc:590] HybridClock initialized: now 1590100138295113 us; error 39 us; skew 500 ppm I0521 22:28:58.296885 751 fs_manager.cc:259] Metadata directory not provided I0521 22:28:58.297034 751 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:28:58.297367 751 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:28:58.300652 751 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/data/instance: uuid: "22eed2da88dd43cbbd818b095d200a61" format_stamp: "Formatted at 2020-05-21 22:28:58 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:58.301492 751 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/wal/instance: uuid: "22eed2da88dd43cbbd818b095d200a61" format_stamp: "Formatted at 2020-05-21 22:28:58 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:58.322190 751 fs_manager.cc:518] Time spent creating directory manager: real 0.020s user 0.018s sys 0.004s I0521 22:28:58.330296 751 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:28:58.330446 751 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/wal uuid: "22eed2da88dd43cbbd818b095d200a61" format_stamp: "Formatted at 2020-05-21 22:28:58 on dist-test-slave-dist-test-slave-jp1g" I0521 22:28:58.330719 751 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0521 22:28:58.506335 750 raft_consensus.cc:500] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528bb2d9313a499aaa0182c4e5d3606d" member_type: VOTER } I0521 22:28:58.507540 750 leader_election.cc:302] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 528bb2d9313a499aaa0182c4e5d3606d; no voters: I0521 22:28:58.509757 750 leader_election.cc:288] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:28:58.510125 765 raft_consensus.cc:2721] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:28:58.938477 765 raft_consensus.cc:683] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [term 1 LEADER]: Becoming Leader. State: Replica: 528bb2d9313a499aaa0182c4e5d3606d, State: Running, Role: LEADER I0521 22:28:58.939661 765 consensus_queue.cc:227] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528bb2d9313a499aaa0182c4e5d3606d" member_type: VOTER } I0521 22:28:58.958058 767 sys_catalog.cc:357] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 528bb2d9313a499aaa0182c4e5d3606d. Latest consensus state: current_term: 1 leader_uuid: "528bb2d9313a499aaa0182c4e5d3606d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528bb2d9313a499aaa0182c4e5d3606d" member_type: VOTER } } I0521 22:28:58.959336 766 sys_catalog.cc:357] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "528bb2d9313a499aaa0182c4e5d3606d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "528bb2d9313a499aaa0182c4e5d3606d" member_type: VOTER } } I0521 22:28:58.960909 766 sys_catalog.cc:360] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [sys.catalog]: This master's current role is: LEADER I0521 22:28:58.960597 750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [sys.catalog]: configured and running, proceeding with master startup. I0521 22:28:58.965080 767 sys_catalog.cc:360] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d [sys.catalog]: This master's current role is: LEADER I0521 22:28:58.970532 769 catalog_manager.cc:1131] Loading table and tablet metadata into memory... I0521 22:28:59.001639 769 catalog_manager.cc:1142] Initializing Kudu internal certificate authority... I0521 22:28:59.108660 769 catalog_manager.cc:1026] Generated new certificate authority record I0521 22:28:59.111214 769 catalog_manager.cc:1151] Loading token signing keys... I0521 22:28:59.148150 769 catalog_manager.cc:4538] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: Generated new TSK 0 I0521 22:28:59.149901 769 catalog_manager.cc:1161] Initializing in-progress tserver states... W0521 22:28:59.210136 751 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.853s user 0.317s sys 0.531s W0521 22:28:59.210526 751 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.853s user 0.318s sys 0.531s I0521 22:28:59.283730 751 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:28:59.289932 751 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:28:59.290100 751 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:28:59.431108 751 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.193:37837 I0521 22:28:59.434352 751 webserver.cc:381] Webserver started at http://127.0.89.193:38971/ using document root and password file I0521 22:28:59.435217 751 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenDefaultParameters.1590100104622499-359/minicluster-data/ts-0/data/info.pb I0521 22:28:59.438871 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver as pid 751 I0521 22:28:59.472235 883 heartbeater.cc:325] Connected to a master server at 127.0.89.254:37037 I0521 22:28:59.472551 883 heartbeater.cc:416] Registering TS with master... I0521 22:28:59.473376 883 heartbeater.cc:465] Master 127.0.89.254:37037 requested a full tablet report, sending... I0521 22:28:59.475888 717 ts_manager.cc:191] Registered new tserver with Master: 22eed2da88dd43cbbd818b095d200a61 (127.0.89.193:37837) I0521 22:28:59.477421 359 external_mini_cluster.cc:699] 1 TS(s) registered with all masters I0521 22:28:59.478543 717 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.89.193:59063 W0521 22:28:59.757871 889 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:28:59.758303 889 flags.cc:403] Enabled unsafe flag: --never_fsync=true I0521 22:29:00.482134 883 heartbeater.cc:457] Master 127.0.89.254:37037 was elected leader, sending a full tablet report... W0521 22:29:00.715901 889 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.909s user 0.410s sys 0.496s W0521 22:29:00.716219 889 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.910s user 0.412s sys 0.496s I0521 22:29:00.777909 717 catalog_manager.cc:1501] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55448: name: "default.loadgen_auto_a5e4ccca01c34a7b95696aa74187d360" schema { columns { name: "key" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 8 seed: 0 } range_schema { columns { name: "key" } } } I0521 22:29:00.861421 818 tablet_service.cc:1232] Processing CreateTablet for tablet d9da032dbe4a4d5e99d8f8611b4f7975 (table=default.loadgen_auto_a5e4ccca01c34a7b95696aa74187d360 [id=3b49c397961948dbbd8b8ff4618fb27b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0521 22:29:00.868067 818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9da032dbe4a4d5e99d8f8611b4f7975. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:00.863430 820 tablet_service.cc:1232] Processing CreateTablet for tablet 97407ed9a29447389a0d5996478ec095 (table=default.loadgen_auto_a5e4ccca01c34a7b95696aa74187d360 [id=3b49c397961948dbbd8b8ff4618fb27b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0521 22:29:00.918082 820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97407ed9a29447389a0d5996478ec095. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:00.866396 816 tablet_service.cc:1232] Processing CreateTablet for tablet 1e2b697a16ce4a509e9710707ba29599 (table=default.loadgen_auto_a5e4ccca01c34a7b95696aa74187d360 [id=3b49c397961948dbbd8b8ff4618fb27b]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED I0521 22:29:00.924731 816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2b697a16ce4a509e9710707ba29599. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:00.866771 819 tablet_service.cc:1232] Processing CreateTablet for tablet dd718edca2b142bcbd22b058976a7ce8 (table=default.loadgen_auto_a5e4ccca01c34a7b95696aa74187d360 [id=3b49c397961948dbbd8b8ff4618fb27b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0521 22:29:00.939553 819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd718edca2b142bcbd22b058976a7ce8. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:00.868875 814 tablet_service.cc:1232] Processing CreateTablet for tablet 026dd6c0a23b4f32b99b761cce35e0e5 (table=default.loadgen_auto_a5e4ccca01c34a7b95696aa74187d360 [id=3b49c397961948dbbd8b8ff4618fb27b]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED I0521 22:29:01.055836 814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 026dd6c0a23b4f32b99b761cce35e0e5. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:00.871291 813 tablet_service.cc:1232] Processing CreateTablet for tablet efac2f865d364295860e193ae88a3617 (table=default.loadgen_auto_a5e4ccca01c34a7b95696aa74187d360 [id=3b49c397961948dbbd8b8ff4618fb27b]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED I0521 22:29:01.072260 813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efac2f865d364295860e193ae88a3617. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:00.872365 815 tablet_service.cc:1232] Processing CreateTablet for tablet c50213863a5946c5acfeb8d31410a77d (table=default.loadgen_auto_a5e4ccca01c34a7b95696aa74187d360 [id=3b49c397961948dbbd8b8ff4618fb27b]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED I0521 22:29:01.076634 815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c50213863a5946c5acfeb8d31410a77d. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:00.872457 817 tablet_service.cc:1232] Processing CreateTablet for tablet 0e1c6fb1311743818390cc85decc6d22 (table=default.loadgen_auto_a5e4ccca01c34a7b95696aa74187d360 [id=3b49c397961948dbbd8b8ff4618fb27b]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED I0521 22:29:01.102651 817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e1c6fb1311743818390cc85decc6d22. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:01.117307 908 tablet_bootstrap.cc:440] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61: Bootstrap starting. I0521 22:29:01.133571 908 tablet_bootstrap.cc:596] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61: No blocks or log segments found. Creating new log. I0521 22:29:01.135027 908 log.cc:819] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61: Log is configured to *not* fsync() on all Append() calls I0521 22:29:01.139703 908 tablet_bootstrap.cc:440] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61: No bootstrap required, opened a new log I0521 22:29:01.140081 908 ts_tablet_manager.cc:1115] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.007s I0521 22:29:01.153770 908 raft_consensus.cc:342] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:01.154318 908 raft_consensus.cc:368] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:01.154570 908 raft_consensus.cc:722] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Initialized, Role: FOLLOWER I0521 22:29:01.155262 908 consensus_queue.cc:250] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:01.155875 908 raft_consensus.cc:382] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:01.156222 908 raft_consensus.cc:478] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:01.156469 908 raft_consensus.cc:2975] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:01.306141 908 raft_consensus.cc:500] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:01.307126 908 leader_election.cc:302] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22eed2da88dd43cbbd818b095d200a61; no voters: I0521 22:29:01.310854 908 leader_election.cc:288] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:01.311408 910 raft_consensus.cc:2721] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:01.315780 908 ts_tablet_manager.cc:1144] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61: Time spent starting tablet: real 0.176s user 0.012s sys 0.016s I0521 22:29:01.316686 910 raft_consensus.cc:683] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [term 1 LEADER]: Becoming Leader. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Running, Role: LEADER I0521 22:29:01.317163 908 tablet_bootstrap.cc:440] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61: Bootstrap starting. I0521 22:29:01.318672 910 consensus_queue.cc:227] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:01.325124 908 tablet_bootstrap.cc:596] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61: No blocks or log segments found. Creating new log. I0521 22:29:01.329876 717 catalog_manager.cc:4202] T d9da032dbe4a4d5e99d8f8611b4f7975 P 22eed2da88dd43cbbd818b095d200a61 reported cstate change: term changed from 0 to 1, leader changed from to 22eed2da88dd43cbbd818b095d200a61 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "22eed2da88dd43cbbd818b095d200a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } health_report { overall_health: HEALTHY } } } I0521 22:29:01.337992 908 tablet_bootstrap.cc:440] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61: No bootstrap required, opened a new log I0521 22:29:01.338282 908 ts_tablet_manager.cc:1115] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.000s I0521 22:29:01.340600 908 raft_consensus.cc:342] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:01.340960 908 raft_consensus.cc:368] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:01.341138 908 raft_consensus.cc:722] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Initialized, Role: FOLLOWER I0521 22:29:01.341591 908 consensus_queue.cc:250] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:01.342025 908 raft_consensus.cc:382] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:01.342232 908 raft_consensus.cc:478] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:01.342499 908 raft_consensus.cc:2975] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Advancing to term 1 W0521 22:29:02.129524 702 debug-util.cc:402] Leaking SignalData structure 0x7b080002b7a0 after lost signal to thread 689 W0521 22:29:02.129973 702 debug-util.cc:402] Leaking SignalData structure 0x7b080002b5e0 after lost signal to thread 748 W0521 22:29:02.132236 908 consensus_meta.cc:220] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61: Time spent flushing consensus metadata: real 0.789s user 0.000s sys 0.005s I0521 22:29:02.132668 908 raft_consensus.cc:500] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:02.133455 908 leader_election.cc:302] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22eed2da88dd43cbbd818b095d200a61; no voters: I0521 22:29:02.136368 908 leader_election.cc:288] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:02.136613 919 raft_consensus.cc:2721] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:02.141232 908 ts_tablet_manager.cc:1144] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61: Time spent starting tablet: real 0.803s user 0.006s sys 0.013s W0521 22:29:02.141505 908 ts_tablet_manager.cc:1174] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61: Tablet startup took 803ms W0521 22:29:02.141734 908 ts_tablet_manager.cc:1176] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61: Trace: 0521 22:29:00.863582 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:00.863777 (+ 195us) service_pool.cc:223] Handling call 0521 22:29:00.924219 (+ 60442us) ts_tablet_manager.cc:466] Acquired tablet manager lock 0521 22:29:00.924328 (+ 109us) ts_tablet_manager.cc:479] Creating new metadata... 0521 22:29:00.933060 (+ 8732us) tablet_metadata.cc:585] Metadata flushed 0521 22:29:00.943272 (+ 10212us) tablet_replica.cc:153] Creating consensus instance 0521 22:29:00.944735 (+ 1463us) inbound_call.cc:163] Queueing success response 0521 22:29:01.316421 (+371686us) ts_tablet_manager.cc:1093] Bootstrapping tablet 0521 22:29:01.338453 (+ 22032us) ts_tablet_manager.cc:1145] Starting tablet replica 0521 22:29:01.339422 (+ 969us) tablet_replica.cc:206] Starting instrumentation 0521 22:29:01.339766 (+ 344us) tablet_replica.cc:225] Starting consensus Metrics: {"raft.queue_time_us":9991,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":186,"tablet-open.queue_time_us":371875,"thread_start_us":5717,"threads_started":3} I0521 22:29:02.144059 908 tablet_bootstrap.cc:440] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61: Bootstrap starting. I0521 22:29:02.149317 908 tablet_bootstrap.cc:596] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61: No blocks or log segments found. Creating new log. I0521 22:29:02.155190 908 tablet_bootstrap.cc:440] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61: No bootstrap required, opened a new log I0521 22:29:02.155647 908 ts_tablet_manager.cc:1115] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s I0521 22:29:02.158334 908 raft_consensus.cc:342] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:02.159132 919 raft_consensus.cc:683] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [term 1 LEADER]: Becoming Leader. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Running, Role: LEADER I0521 22:29:02.159343 908 raft_consensus.cc:368] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:02.159655 908 raft_consensus.cc:722] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Initialized, Role: FOLLOWER I0521 22:29:02.159838 919 consensus_queue.cc:227] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:02.160619 908 consensus_queue.cc:250] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:02.161373 908 raft_consensus.cc:382] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:02.161809 908 raft_consensus.cc:478] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:02.162163 908 raft_consensus.cc:2975] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:02.167161 717 catalog_manager.cc:4202] T 1e2b697a16ce4a509e9710707ba29599 P 22eed2da88dd43cbbd818b095d200a61 reported cstate change: term changed from 0 to 1, leader changed from to 22eed2da88dd43cbbd818b095d200a61 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "22eed2da88dd43cbbd818b095d200a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } health_report { overall_health: HEALTHY } } } W0521 22:29:02.941017 908 consensus_meta.cc:220] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61: Time spent flushing consensus metadata: real 0.778s user 0.003s sys 0.002s I0521 22:29:02.941275 908 raft_consensus.cc:500] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:02.941802 908 leader_election.cc:302] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22eed2da88dd43cbbd818b095d200a61; no voters: I0521 22:29:02.943224 908 leader_election.cc:288] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:02.943518 928 raft_consensus.cc:2721] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:02.946138 908 ts_tablet_manager.cc:1144] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61: Time spent starting tablet: real 0.790s user 0.008s sys 0.006s W0521 22:29:02.946447 908 ts_tablet_manager.cc:1174] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61: Tablet startup took 790ms W0521 22:29:02.946676 908 ts_tablet_manager.cc:1176] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61: Trace: 0521 22:29:00.856690 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:00.856864 (+ 174us) service_pool.cc:223] Handling call 0521 22:29:00.917563 (+ 60699us) ts_tablet_manager.cc:466] Acquired tablet manager lock 0521 22:29:00.917650 (+ 87us) ts_tablet_manager.cc:479] Creating new metadata... 0521 22:29:00.926531 (+ 8881us) tablet_metadata.cc:585] Metadata flushed 0521 22:29:00.945656 (+ 19125us) tablet_replica.cc:153] Creating consensus instance 0521 22:29:00.946252 (+ 596us) inbound_call.cc:163] Queueing success response 0521 22:29:02.143804 (+1197552us) ts_tablet_manager.cc:1093] Bootstrapping tablet 0521 22:29:02.155943 (+ 12139us) ts_tablet_manager.cc:1145] Starting tablet replica 0521 22:29:02.156467 (+ 524us) tablet_replica.cc:206] Starting instrumentation 0521 22:29:02.157061 (+ 594us) tablet_replica.cc:225] Starting consensus Metrics: {"raft.queue_time_us":5228,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":686,"tablet-open.queue_time_us":1197869,"thread_start_us":2510,"threads_started":3} W0521 22:29:02.948531 884 tablet.cc:1820] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61: Can't schedule compaction. Clean time has not been advanced past its initial value. I0521 22:29:02.949168 908 tablet_bootstrap.cc:440] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61: Bootstrap starting. I0521 22:29:02.952391 928 raft_consensus.cc:683] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [term 1 LEADER]: Becoming Leader. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Running, Role: LEADER I0521 22:29:02.953022 928 consensus_queue.cc:227] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:02.959028 908 tablet_bootstrap.cc:596] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61: No blocks or log segments found. Creating new log. I0521 22:29:02.960505 717 catalog_manager.cc:4202] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 reported cstate change: term changed from 0 to 1, leader changed from to 22eed2da88dd43cbbd818b095d200a61 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "22eed2da88dd43cbbd818b095d200a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } health_report { overall_health: HEALTHY } } } I0521 22:29:02.967530 908 tablet_bootstrap.cc:440] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61: No bootstrap required, opened a new log I0521 22:29:02.967828 908 ts_tablet_manager.cc:1115] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.013s I0521 22:29:02.969803 908 raft_consensus.cc:342] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:02.970309 908 raft_consensus.cc:368] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:02.970580 908 raft_consensus.cc:722] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Initialized, Role: FOLLOWER I0521 22:29:02.971382 908 consensus_queue.cc:250] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:02.972122 908 raft_consensus.cc:382] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:02.972443 908 raft_consensus.cc:478] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:02.973603 908 raft_consensus.cc:2975] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Advancing to term 1 W0521 22:29:03.779189 908 consensus_meta.cc:220] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61: Time spent flushing consensus metadata: real 0.805s user 0.003s sys 0.003s I0521 22:29:03.779525 908 raft_consensus.cc:500] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.780249 908 leader_election.cc:302] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22eed2da88dd43cbbd818b095d200a61; no voters: I0521 22:29:03.781239 908 leader_election.cc:288] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:03.781495 937 raft_consensus.cc:2721] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:03.782353 937 raft_consensus.cc:683] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [term 1 LEADER]: Becoming Leader. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Running, Role: LEADER I0521 22:29:03.782948 937 consensus_queue.cc:227] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.784921 908 ts_tablet_manager.cc:1144] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61: Time spent starting tablet: real 0.817s user 0.009s sys 0.007s W0521 22:29:03.785135 908 ts_tablet_manager.cc:1174] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61: Tablet startup took 817ms W0521 22:29:03.785331 908 ts_tablet_manager.cc:1176] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61: Trace: 0521 22:29:00.857039 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:00.857168 (+ 129us) service_pool.cc:223] Handling call 0521 22:29:00.938823 (+ 81655us) ts_tablet_manager.cc:466] Acquired tablet manager lock 0521 22:29:00.938908 (+ 85us) ts_tablet_manager.cc:479] Creating new metadata... 0521 22:29:00.948634 (+ 9726us) tablet_metadata.cc:585] Metadata flushed 0521 22:29:00.957769 (+ 9135us) tablet_replica.cc:153] Creating consensus instance 0521 22:29:00.958446 (+ 677us) inbound_call.cc:163] Queueing success response 0521 22:29:02.948902 (+1990456us) ts_tablet_manager.cc:1093] Bootstrapping tablet 0521 22:29:02.968002 (+ 19100us) ts_tablet_manager.cc:1145] Starting tablet replica 0521 22:29:02.968338 (+ 336us) tablet_replica.cc:206] Starting instrumentation 0521 22:29:02.968661 (+ 323us) tablet_replica.cc:225] Starting consensus 0521 22:29:03.784075 (+815414us) log.cc:840] Serialized 26 byte log entry Metrics: {"raft.queue_time_us":3348,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":798,"tablet-open.queue_time_us":1990829,"thread_start_us":2966,"threads_started":4} I0521 22:29:03.787286 908 tablet_bootstrap.cc:440] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61: Bootstrap starting. I0521 22:29:03.791777 717 catalog_manager.cc:4202] T dd718edca2b142bcbd22b058976a7ce8 P 22eed2da88dd43cbbd818b095d200a61 reported cstate change: term changed from 0 to 1, leader changed from to 22eed2da88dd43cbbd818b095d200a61 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "22eed2da88dd43cbbd818b095d200a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } health_report { overall_health: HEALTHY } } } I0521 22:29:03.792593 908 tablet_bootstrap.cc:596] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61: No blocks or log segments found. Creating new log. I0521 22:29:03.800877 908 tablet_bootstrap.cc:440] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61: No bootstrap required, opened a new log I0521 22:29:03.801285 908 ts_tablet_manager.cc:1115] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.004s I0521 22:29:03.803558 908 raft_consensus.cc:342] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.804108 908 raft_consensus.cc:368] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:03.804370 908 raft_consensus.cc:722] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Initialized, Role: FOLLOWER I0521 22:29:03.804968 908 consensus_queue.cc:250] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.805543 908 raft_consensus.cc:382] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:03.805820 908 raft_consensus.cc:478] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:03.806056 908 raft_consensus.cc:2975] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:03.811376 908 raft_consensus.cc:500] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.811931 908 leader_election.cc:302] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22eed2da88dd43cbbd818b095d200a61; no voters: I0521 22:29:03.812472 908 leader_election.cc:288] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:03.812599 937 raft_consensus.cc:2721] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:03.814786 937 raft_consensus.cc:683] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [term 1 LEADER]: Becoming Leader. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Running, Role: LEADER I0521 22:29:03.815454 908 ts_tablet_manager.cc:1144] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s I0521 22:29:03.815378 937 consensus_queue.cc:227] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.816385 908 tablet_bootstrap.cc:440] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61: Bootstrap starting. I0521 22:29:03.822146 908 tablet_bootstrap.cc:596] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61: No blocks or log segments found. Creating new log. I0521 22:29:03.829733 908 tablet_bootstrap.cc:440] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61: No bootstrap required, opened a new log I0521 22:29:03.830081 908 ts_tablet_manager.cc:1115] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s I0521 22:29:03.829783 717 catalog_manager.cc:4202] T 026dd6c0a23b4f32b99b761cce35e0e5 P 22eed2da88dd43cbbd818b095d200a61 reported cstate change: term changed from 0 to 1, leader changed from to 22eed2da88dd43cbbd818b095d200a61 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "22eed2da88dd43cbbd818b095d200a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } health_report { overall_health: HEALTHY } } } I0521 22:29:03.832113 908 raft_consensus.cc:342] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.832624 908 raft_consensus.cc:368] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:03.832862 908 raft_consensus.cc:722] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Initialized, Role: FOLLOWER I0521 22:29:03.833349 908 consensus_queue.cc:250] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.833895 908 raft_consensus.cc:382] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:03.834123 908 raft_consensus.cc:478] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:03.834360 908 raft_consensus.cc:2975] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:03.839583 908 raft_consensus.cc:500] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.840184 908 leader_election.cc:302] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22eed2da88dd43cbbd818b095d200a61; no voters: I0521 22:29:03.840648 908 leader_election.cc:288] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:03.840792 937 raft_consensus.cc:2721] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:03.841181 937 raft_consensus.cc:683] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [term 1 LEADER]: Becoming Leader. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Running, Role: LEADER I0521 22:29:03.841771 937 consensus_queue.cc:227] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.843649 908 ts_tablet_manager.cc:1144] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s I0521 22:29:03.844364 908 tablet_bootstrap.cc:440] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61: Bootstrap starting. I0521 22:29:03.848310 717 catalog_manager.cc:4202] T efac2f865d364295860e193ae88a3617 P 22eed2da88dd43cbbd818b095d200a61 reported cstate change: term changed from 0 to 1, leader changed from to 22eed2da88dd43cbbd818b095d200a61 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "22eed2da88dd43cbbd818b095d200a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } health_report { overall_health: HEALTHY } } } I0521 22:29:03.848991 908 tablet_bootstrap.cc:596] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61: No blocks or log segments found. Creating new log. I0521 22:29:03.854046 908 tablet_bootstrap.cc:440] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61: No bootstrap required, opened a new log I0521 22:29:03.854334 908 ts_tablet_manager.cc:1115] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s I0521 22:29:03.856020 908 raft_consensus.cc:342] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.856438 908 raft_consensus.cc:368] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:03.856611 908 raft_consensus.cc:722] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Initialized, Role: FOLLOWER I0521 22:29:03.857031 908 consensus_queue.cc:250] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.857420 908 raft_consensus.cc:382] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:03.857587 908 raft_consensus.cc:478] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:03.857777 908 raft_consensus.cc:2975] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:03.861145 908 raft_consensus.cc:500] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.861557 908 leader_election.cc:302] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22eed2da88dd43cbbd818b095d200a61; no voters: I0521 22:29:03.861913 908 leader_election.cc:288] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:03.862040 937 raft_consensus.cc:2721] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:03.862443 937 raft_consensus.cc:683] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [term 1 LEADER]: Becoming Leader. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Running, Role: LEADER I0521 22:29:03.863088 908 ts_tablet_manager.cc:1144] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s I0521 22:29:03.862948 937 consensus_queue.cc:227] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.863621 908 tablet_bootstrap.cc:440] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61: Bootstrap starting. I0521 22:29:03.866837 908 tablet_bootstrap.cc:596] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61: No blocks or log segments found. Creating new log. I0521 22:29:03.872864 717 catalog_manager.cc:4202] T c50213863a5946c5acfeb8d31410a77d P 22eed2da88dd43cbbd818b095d200a61 reported cstate change: term changed from 0 to 1, leader changed from to 22eed2da88dd43cbbd818b095d200a61 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "22eed2da88dd43cbbd818b095d200a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } health_report { overall_health: HEALTHY } } } I0521 22:29:03.875278 908 tablet_bootstrap.cc:440] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61: No bootstrap required, opened a new log I0521 22:29:03.875567 908 ts_tablet_manager.cc:1115] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s I0521 22:29:03.877315 908 raft_consensus.cc:342] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.877789 908 raft_consensus.cc:368] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:03.877981 908 raft_consensus.cc:722] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Initialized, Role: FOLLOWER I0521 22:29:03.878449 908 consensus_queue.cc:250] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.879021 908 raft_consensus.cc:382] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:03.879189 908 raft_consensus.cc:478] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:03.879401 908 raft_consensus.cc:2975] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:03.882751 908 raft_consensus.cc:500] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.883162 908 leader_election.cc:302] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22eed2da88dd43cbbd818b095d200a61; no voters: I0521 22:29:03.883534 908 leader_election.cc:288] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:03.883745 937 raft_consensus.cc:2721] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:03.884404 937 raft_consensus.cc:683] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [term 1 LEADER]: Becoming Leader. State: Replica: 22eed2da88dd43cbbd818b095d200a61, State: Running, Role: LEADER I0521 22:29:03.885161 908 ts_tablet_manager.cc:1144] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s I0521 22:29:03.885044 937 consensus_queue.cc:227] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } } I0521 22:29:03.890064 717 catalog_manager.cc:4202] T 0e1c6fb1311743818390cc85decc6d22 P 22eed2da88dd43cbbd818b095d200a61 reported cstate change: term changed from 0 to 1, leader changed from to 22eed2da88dd43cbbd818b095d200a61 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "22eed2da88dd43cbbd818b095d200a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22eed2da88dd43cbbd818b095d200a61" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37837 } health_report { overall_health: HEALTHY } } } Using auto-created table 'default.loadgen_auto_a5e4ccca01c34a7b95696aa74187d360' INSERT report rows total: 2000 time total: 771.027 ms time per row: 0.385514 ms Dropping auto-created table 'default.loadgen_auto_a5e4ccca01c34a7b95696aa74187d360' I0521 22:29:04.915191 714 catalog_manager.cc:1999] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55448: table { table_name: "default.loadgen_auto_a5e4ccca01c34a7b95696aa74187d360" } modify_external_catalogs: true I0521 22:29:04.930640 714 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: Sending DeleteTablet for 1 replicas of tablet 97407ed9a29447389a0d5996478ec095 I0521 22:29:04.931238 714 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: Sending DeleteTablet for 1 replicas of tablet dd718edca2b142bcbd22b058976a7ce8 I0521 22:29:04.931782 714 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: Sending DeleteTablet for 1 replicas of tablet d9da032dbe4a4d5e99d8f8611b4f7975 I0521 22:29:04.932179 817 tablet_service.cc:1276] Processing DeleteTablet for tablet 97407ed9a29447389a0d5996478ec095 with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:04 UTC) from {username='slave'} at 127.0.0.1:37064 I0521 22:29:04.932761 815 tablet_service.cc:1276] Processing DeleteTablet for tablet dd718edca2b142bcbd22b058976a7ce8 with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:04 UTC) from {username='slave'} at 127.0.0.1:37064 I0521 22:29:04.932924 714 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: Sending DeleteTablet for 1 replicas of tablet 0e1c6fb1311743818390cc85decc6d22 I0521 22:29:04.933522 714 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: Sending DeleteTablet for 1 replicas of tablet 1e2b697a16ce4a509e9710707ba29599 I0521 22:29:04.934155 714 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: Sending DeleteTablet for 1 replicas of tablet c50213863a5946c5acfeb8d31410a77d I0521 22:29:04.934803 714 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: Sending DeleteTablet for 1 replicas of tablet 026dd6c0a23b4f32b99b761cce35e0e5 I0521 22:29:04.935406 813 tablet_service.cc:1276] Processing DeleteTablet for tablet d9da032dbe4a4d5e99d8f8611b4f7975 with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:04 UTC) from {username='slave'} at 127.0.0.1:37064 I0521 22:29:04.935773 982 tablet_replica.cc:277] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61: stopping tablet replica I0521 22:29:04.936234 818 tablet_service.cc:1276] Processing DeleteTablet for tablet 1e2b697a16ce4a509e9710707ba29599 with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:04 UTC) from {username='slave'} at 127.0.0.1:37064 I0521 22:29:04.936229 815 tablet_service.cc:1276] Processing DeleteTablet for tablet 0e1c6fb1311743818390cc85decc6d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:04 UTC) from {username='slave'} at 127.0.0.1:37064 I0521 22:29:04.936553 982 raft_consensus.cc:2186] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [term 1 LEADER]: Raft consensus shutting down. I0521 22:29:04.936962 982 raft_consensus.cc:2213] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61 [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:29:04.937536 714 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 528bb2d9313a499aaa0182c4e5d3606d: Sending DeleteTablet for 1 replicas of tablet efac2f865d364295860e193ae88a3617 I0521 22:29:04.938730 815 tablet_service.cc:1276] Processing DeleteTablet for tablet 026dd6c0a23b4f32b99b761cce35e0e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:04 UTC) from {username='slave'} at 127.0.0.1:37064 I0521 22:29:04.939064 818 tablet_service.cc:1276] Processing DeleteTablet for tablet efac2f865d364295860e193ae88a3617 with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:04 UTC) from {username='slave'} at 127.0.0.1:37064 I0521 22:29:04.938541 813 tablet_service.cc:1276] Processing DeleteTablet for tablet c50213863a5946c5acfeb8d31410a77d with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:04 UTC) from {username='slave'} at 127.0.0.1:37064 I0521 22:29:04.941203 982 ts_tablet_manager.cc:1476] T 97407ed9a29447389a0d5996478ec095 P 22eed2da88dd43cbbd818b095d200a61: Deleting tablet data with delete state TABLET_DATA_DELETED I0521 22:29:05.965721 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver with pid 751 I0521 22:29:06.293896 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master with pid 688 W0521 22:29:06.293926 701 catalog_manager.cc:3321] TS 22eed2da88dd43cbbd818b095d200a61 (127.0.89.193:37837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet efac2f865d364295860e193ae88a3617: Network error: recv got EOF from 127.0.89.193:37837 (error 108) 2020-05-21T22:29:06Z chronyd exiting [ OK ] ToolTest.TestLoadgenDefaultParameters (9359 ms) [ RUN ] ToolTest.TestLoadgenManualFlush 2020-05-21T22:29:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-05-21T22:29:06Z Disabled control of system clock I0521 22:29:06.359498 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --ipki_ca_key_size=768 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:45857 --webserver_interface=127.0.89.254 --webserver_port=0 --builtin_ntp_servers=127.0.89.212:43663 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:29:06.503576 986 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:29:06.504160 986 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:29:06.504627 986 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:29:06.516865 986 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=768 W0521 22:29:06.517036 986 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:29:06.517233 986 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0521 22:29:06.517424 986 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0521 22:29:06.535710 986 master_runner.cc:93] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:43663 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/wal --ipki_ca_key_size=768 --ipki_server_key_size=768 --openssl_security_level_override=0 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:45857 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.89.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:29:06.536581 986 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:29:06.538009 986 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:29:06.552683 986 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:29:06.555938 986 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:29:06.557274 986 hybrid_clock.cc:590] HybridClock initialized: now 1590100146557236 us; error 58 us; skew 500 ppm I0521 22:29:06.559031 986 fs_manager.cc:259] Metadata directory not provided I0521 22:29:06.559204 986 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:29:06.559543 986 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:29:06.562616 986 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/data/instance: uuid: "796631a6c5bf42fd89e34b92b884cc0f" format_stamp: "Formatted at 2020-05-21 22:29:06 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:06.563431 986 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/wal/instance: uuid: "796631a6c5bf42fd89e34b92b884cc0f" format_stamp: "Formatted at 2020-05-21 22:29:06 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:06.583232 986 fs_manager.cc:518] Time spent creating directory manager: real 0.019s user 0.022s sys 0.000s I0521 22:29:06.591326 986 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:29:06.591485 986 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/wal uuid: "796631a6c5bf42fd89e34b92b884cc0f" format_stamp: "Formatted at 2020-05-21 22:29:06 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:06.591759 986 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:29:07.372006 986 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.754s user 0.345s sys 0.389s W0521 22:29:07.372269 986 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.754s user 0.345s sys 0.389s I0521 22:29:07.386659 986 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:29:07.454813 986 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.254:45857 I0521 22:29:07.458307 986 webserver.cc:381] Webserver started at http://127.0.89.254:33603/ using document root and password file I0521 22:29:07.459277 986 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/master-0/data/info.pb I0521 22:29:07.463256 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master as pid 986 I0521 22:29:07.464236 1048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:07.465647 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --rpc_bind_addresses=127.0.89.193:0 --local_ip_for_outbound_sockets=127.0.89.193 --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:45857 --builtin_ntp_servers=127.0.89.212:43663 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} I0521 22:29:07.492439 1048 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f: Bootstrap starting. I0521 22:29:07.498879 1048 tablet_bootstrap.cc:596] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f: No blocks or log segments found. Creating new log. I0521 22:29:07.501101 1048 log.cc:819] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f: Log is configured to *not* fsync() on all Append() calls I0521 22:29:07.507138 1048 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f: No bootstrap required, opened a new log I0521 22:29:07.532745 1048 raft_consensus.cc:342] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796631a6c5bf42fd89e34b92b884cc0f" member_type: VOTER } I0521 22:29:07.533361 1048 raft_consensus.cc:368] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:07.533617 1048 raft_consensus.cc:722] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796631a6c5bf42fd89e34b92b884cc0f, State: Initialized, Role: FOLLOWER I0521 22:29:07.534446 1048 consensus_queue.cc:250] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796631a6c5bf42fd89e34b92b884cc0f" member_type: VOTER } I0521 22:29:07.535040 1048 raft_consensus.cc:382] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:07.535339 1048 raft_consensus.cc:478] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:07.535663 1048 raft_consensus.cc:2975] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:07.540112 1048 raft_consensus.cc:500] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796631a6c5bf42fd89e34b92b884cc0f" member_type: VOTER } I0521 22:29:07.540644 1048 leader_election.cc:302] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 796631a6c5bf42fd89e34b92b884cc0f; no voters: I0521 22:29:07.542309 1048 leader_election.cc:288] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [CANDIDATE]: Term 1 election: Requested vote from peers WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:29:07.598482 1049 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:29:07.598922 1049 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:29:07.599462 1049 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:29:07.610006 1049 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:29:07.610914 1049 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.89.193 I0521 22:29:07.627918 1049 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:43663 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/wal --ipki_server_key_size=768 --openssl_security_level_override=0 --rpc_bind_addresses=127.0.89.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:45857 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.89.193 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:29:07.628924 1049 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:29:07.630365 1049 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:29:07.646476 1049 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:29:07.649299 1049 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:29:07.650630 1049 hybrid_clock.cc:590] HybridClock initialized: now 1590100147650595 us; error 49 us; skew 500 ppm I0521 22:29:07.652460 1049 fs_manager.cc:259] Metadata directory not provided I0521 22:29:07.652621 1049 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:29:07.652966 1049 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:29:07.656106 1049 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/data/instance: uuid: "343052a5d259480dbf4599078770bfe1" format_stamp: "Formatted at 2020-05-21 22:29:07 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:07.656944 1049 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/wal/instance: uuid: "343052a5d259480dbf4599078770bfe1" format_stamp: "Formatted at 2020-05-21 22:29:07 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:07.683764 1049 fs_manager.cc:518] Time spent creating directory manager: real 0.026s user 0.021s sys 0.001s I0521 22:29:07.692076 1049 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:29:07.692234 1049 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/wal uuid: "343052a5d259480dbf4599078770bfe1" format_stamp: "Formatted at 2020-05-21 22:29:07 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:07.692540 1049 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0521 22:29:07.542667 1051 raft_consensus.cc:2721] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:07.769850 1051 raft_consensus.cc:683] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [term 1 LEADER]: Becoming Leader. State: Replica: 796631a6c5bf42fd89e34b92b884cc0f, State: Running, Role: LEADER I0521 22:29:07.770900 1051 consensus_queue.cc:227] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796631a6c5bf42fd89e34b92b884cc0f" member_type: VOTER } I0521 22:29:07.779135 1048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [sys.catalog]: configured and running, proceeding with master startup. I0521 22:29:07.782356 1065 sys_catalog.cc:357] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 796631a6c5bf42fd89e34b92b884cc0f. Latest consensus state: current_term: 1 leader_uuid: "796631a6c5bf42fd89e34b92b884cc0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796631a6c5bf42fd89e34b92b884cc0f" member_type: VOTER } } I0521 22:29:07.783146 1065 sys_catalog.cc:360] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [sys.catalog]: This master's current role is: LEADER I0521 22:29:07.782619 1052 sys_catalog.cc:357] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "796631a6c5bf42fd89e34b92b884cc0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796631a6c5bf42fd89e34b92b884cc0f" member_type: VOTER } } I0521 22:29:07.783807 1052 sys_catalog.cc:360] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f [sys.catalog]: This master's current role is: LEADER I0521 22:29:07.785840 1068 catalog_manager.cc:1131] Loading table and tablet metadata into memory... I0521 22:29:07.800469 1068 catalog_manager.cc:1142] Initializing Kudu internal certificate authority... I0521 22:29:07.871320 1068 catalog_manager.cc:1026] Generated new certificate authority record I0521 22:29:07.875003 1068 catalog_manager.cc:1151] Loading token signing keys... I0521 22:29:07.894775 1068 catalog_manager.cc:4538] T 00000000000000000000000000000000 P 796631a6c5bf42fd89e34b92b884cc0f: Generated new TSK 0 I0521 22:29:07.896216 1068 catalog_manager.cc:1161] Initializing in-progress tserver states... W0521 22:29:08.495518 1049 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.778s user 0.289s sys 0.478s W0521 22:29:08.495884 1049 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.779s user 0.289s sys 0.479s I0521 22:29:08.529492 1049 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:29:08.538825 1049 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:29:08.539120 1049 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:29:08.956528 1049 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.193:45719 I0521 22:29:08.964589 1049 webserver.cc:381] Webserver started at http://127.0.89.193:42679/ using document root and password file I0521 22:29:08.965701 1049 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-0/data/info.pb I0521 22:29:08.969755 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver as pid 1049 I0521 22:29:08.972070 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --rpc_bind_addresses=127.0.89.194:0 --local_ip_for_outbound_sockets=127.0.89.194 --webserver_interface=127.0.89.194 --webserver_port=0 --tserver_master_addrs=127.0.89.254:45857 --builtin_ntp_servers=127.0.89.212:43663 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} I0521 22:29:09.015504 1181 heartbeater.cc:325] Connected to a master server at 127.0.89.254:45857 I0521 22:29:09.015841 1181 heartbeater.cc:416] Registering TS with master... I0521 22:29:09.016562 1181 heartbeater.cc:465] Master 127.0.89.254:45857 requested a full tablet report, sending... I0521 22:29:09.018663 1015 ts_manager.cc:191] Registered new tserver with Master: 343052a5d259480dbf4599078770bfe1 (127.0.89.193:45719) I0521 22:29:09.020560 1015 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.89.193:48533 WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:29:09.177631 1185 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:29:09.178231 1185 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:29:09.179066 1185 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:29:09.198398 1185 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:29:09.199884 1185 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.89.194 I0521 22:29:09.230556 1185 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:43663 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/wal --ipki_server_key_size=768 --openssl_security_level_override=0 --rpc_bind_addresses=127.0.89.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/data/info.pb --webserver_interface=127.0.89.194 --webserver_port=0 --tserver_master_addrs=127.0.89.254:45857 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.89.194 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:29:09.231761 1185 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:29:09.233717 1185 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:29:09.260821 1185 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:29:09.265516 1185 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:29:09.266889 1185 hybrid_clock.cc:590] HybridClock initialized: now 1590100149266810 us; error 87 us; skew 500 ppm I0521 22:29:09.268815 1185 fs_manager.cc:259] Metadata directory not provided I0521 22:29:09.268982 1185 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:29:09.269353 1185 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:29:09.272707 1185 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/data/instance: uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" format_stamp: "Formatted at 2020-05-21 22:29:09 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:09.273654 1185 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/wal/instance: uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" format_stamp: "Formatted at 2020-05-21 22:29:09 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:09.305495 1185 fs_manager.cc:518] Time spent creating directory manager: real 0.031s user 0.020s sys 0.012s I0521 22:29:09.313904 1185 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:29:09.314146 1185 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/wal uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" format_stamp: "Formatted at 2020-05-21 22:29:09 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:09.314452 1185 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0521 22:29:10.023569 1181 heartbeater.cc:457] Master 127.0.89.254:45857 was elected leader, sending a full tablet report... W0521 22:29:10.422003 1185 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 1.066s user 0.411s sys 0.605s W0521 22:29:10.422204 1185 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 1.066s user 0.411s sys 0.605s I0521 22:29:10.461632 1185 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:29:10.467905 1185 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:29:10.468156 1185 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:29:10.611158 1185 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.194:46087 I0521 22:29:10.614740 1185 webserver.cc:381] Webserver started at http://127.0.89.194:34851/ using document root and password file I0521 22:29:10.615636 1185 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-1/data/info.pb I0521 22:29:10.617156 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver as pid 1185 I0521 22:29:10.618960 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --rpc_bind_addresses=127.0.89.195:0 --local_ip_for_outbound_sockets=127.0.89.195 --webserver_interface=127.0.89.195 --webserver_port=0 --tserver_master_addrs=127.0.89.254:45857 --builtin_ntp_servers=127.0.89.212:43663 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} I0521 22:29:10.649358 1307 heartbeater.cc:325] Connected to a master server at 127.0.89.254:45857 I0521 22:29:10.649772 1307 heartbeater.cc:416] Registering TS with master... I0521 22:29:10.650499 1307 heartbeater.cc:465] Master 127.0.89.254:45857 requested a full tablet report, sending... I0521 22:29:10.652485 1015 ts_manager.cc:191] Registered new tserver with Master: aa747fc1bcba4e4fb55a9bce3a0c7c37 (127.0.89.194:46087) I0521 22:29:10.654449 1015 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.89.194:51033 WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:29:10.756886 1311 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:29:10.757277 1311 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:29:10.757758 1311 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:29:10.768251 1311 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:29:10.769136 1311 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.89.195 I0521 22:29:10.785708 1311 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:43663 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/wal --ipki_server_key_size=768 --openssl_security_level_override=0 --rpc_bind_addresses=127.0.89.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/data/info.pb --webserver_interface=127.0.89.195 --webserver_port=0 --tserver_master_addrs=127.0.89.254:45857 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.89.195 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:29:10.786620 1311 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:29:10.788060 1311 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:29:10.803423 1311 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:29:10.806653 1311 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:29:10.807927 1311 hybrid_clock.cc:590] HybridClock initialized: now 1590100150807859 us; error 71 us; skew 500 ppm I0521 22:29:10.809670 1311 fs_manager.cc:259] Metadata directory not provided I0521 22:29:10.809845 1311 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:29:10.810192 1311 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:29:10.813504 1311 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/data/instance: uuid: "e4889a5698f944459719d537988dbaea" format_stamp: "Formatted at 2020-05-21 22:29:10 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:10.814410 1311 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/wal/instance: uuid: "e4889a5698f944459719d537988dbaea" format_stamp: "Formatted at 2020-05-21 22:29:10 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:10.835546 1311 fs_manager.cc:518] Time spent creating directory manager: real 0.021s user 0.022s sys 0.000s I0521 22:29:10.843619 1311 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:29:10.843761 1311 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/wal uuid: "e4889a5698f944459719d537988dbaea" format_stamp: "Formatted at 2020-05-21 22:29:10 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:10.844082 1311 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:29:11.607105 1311 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.739s user 0.309s sys 0.428s W0521 22:29:11.607409 1311 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.740s user 0.309s sys 0.428s I0521 22:29:11.639135 1311 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:29:11.645370 1311 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:29:11.645587 1311 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:29:11.657428 1307 heartbeater.cc:457] Master 127.0.89.254:45857 was elected leader, sending a full tablet report... I0521 22:29:11.780596 1311 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.195:35691 I0521 22:29:11.783752 1311 webserver.cc:381] Webserver started at http://127.0.89.195:43255/ using document root and password file I0521 22:29:11.784653 1311 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenManualFlush.1590100104622499-359/minicluster-data/ts-2/data/info.pb I0521 22:29:11.786253 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver as pid 1311 I0521 22:29:11.812394 1433 heartbeater.cc:325] Connected to a master server at 127.0.89.254:45857 I0521 22:29:11.812719 1433 heartbeater.cc:416] Registering TS with master... I0521 22:29:11.813596 1433 heartbeater.cc:465] Master 127.0.89.254:45857 requested a full tablet report, sending... I0521 22:29:11.815485 1015 ts_manager.cc:191] Registered new tserver with Master: e4889a5698f944459719d537988dbaea (127.0.89.195:35691) I0521 22:29:11.817246 1015 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.89.195:41683 I0521 22:29:11.818189 359 external_mini_cluster.cc:699] 3 TS(s) registered with all masters I0521 22:29:11.851701 1015 catalog_manager.cc:1501] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50490: name: "bench_manual_flush" schema { columns { name: "key" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "bool_val" type: BOOL is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int8_val" type: INT8 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int16_val" type: INT16 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int32_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int64_val" type: INT64 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "float_val" type: FLOAT is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "double_val" type: DOUBLE is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "decimal32_val" type: DECIMAL32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 type_attributes { precision: 9 scale: 9 } } columns { name: "decimal64_val" type: DECIMAL64 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 type_attributes { precision: 18 scale: 2 } } columns { name: "decimal128_val" type: DECIMAL128 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 type_attributes { precision: 38 scale: 0 } } columns { name: "unixtime_micros_val" type: UNIXTIME_MICROS is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "binary_val" type: BINARY is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 2 seed: 0 } } W0521 22:29:11.866477 1015 catalog_manager.cc:1666] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table bench_manual_flush in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0521 22:29:11.927067 1244 tablet_service.cc:1232] Processing CreateTablet for tablet db62be61ab374802a1d543ea858f9113 (table=bench_manual_flush [id=9436caed107d45b6aac9e72ee79baf33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0521 22:29:11.929874 1244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db62be61ab374802a1d543ea858f9113. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:11.933924 1369 tablet_service.cc:1232] Processing CreateTablet for tablet e7001908da8140df8082f3243384a351 (table=bench_manual_flush [id=9436caed107d45b6aac9e72ee79baf33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0521 22:29:11.936858 1369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7001908da8140df8082f3243384a351. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:11.930771 1118 tablet_service.cc:1232] Processing CreateTablet for tablet db62be61ab374802a1d543ea858f9113 (table=bench_manual_flush [id=9436caed107d45b6aac9e72ee79baf33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0521 22:29:11.941782 1118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db62be61ab374802a1d543ea858f9113. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:11.933928 1370 tablet_service.cc:1232] Processing CreateTablet for tablet db62be61ab374802a1d543ea858f9113 (table=bench_manual_flush [id=9436caed107d45b6aac9e72ee79baf33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0521 22:29:11.995908 1370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db62be61ab374802a1d543ea858f9113. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:12.007134 1454 tablet_bootstrap.cc:440] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Bootstrap starting. I0521 22:29:11.932466 1117 tablet_service.cc:1232] Processing CreateTablet for tablet e7001908da8140df8082f3243384a351 (table=bench_manual_flush [id=9436caed107d45b6aac9e72ee79baf33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0521 22:29:12.028003 1117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7001908da8140df8082f3243384a351. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:12.037917 1454 tablet_bootstrap.cc:596] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: No blocks or log segments found. Creating new log. I0521 22:29:12.038583 1455 tablet_bootstrap.cc:440] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Bootstrap starting. I0521 22:29:12.039683 1454 log.cc:819] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Log is configured to *not* fsync() on all Append() calls I0521 22:29:12.051631 1455 tablet_bootstrap.cc:596] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: No blocks or log segments found. Creating new log. I0521 22:29:12.054328 1455 log.cc:819] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Log is configured to *not* fsync() on all Append() calls I0521 22:29:12.056092 1454 tablet_bootstrap.cc:440] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: No bootstrap required, opened a new log I0521 22:29:12.056594 1454 ts_tablet_manager.cc:1115] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.011s I0521 22:29:12.064152 1455 tablet_bootstrap.cc:440] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: No bootstrap required, opened a new log I0521 22:29:12.064666 1455 ts_tablet_manager.cc:1115] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.012s I0521 22:29:12.085675 1454 raft_consensus.cc:342] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.086751 1454 raft_consensus.cc:368] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:12.087041 1454 raft_consensus.cc:722] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4889a5698f944459719d537988dbaea, State: Initialized, Role: FOLLOWER I0521 22:29:12.087990 1454 consensus_queue.cc:250] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.093050 1433 heartbeater.cc:457] Master 127.0.89.254:45857 was elected leader, sending a full tablet report... I0521 22:29:12.093480 1454 ts_tablet_manager.cc:1144] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Time spent starting tablet: real 0.037s user 0.026s sys 0.007s I0521 22:29:12.094933 1454 tablet_bootstrap.cc:440] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Bootstrap starting. I0521 22:29:12.096691 1455 raft_consensus.cc:342] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.097707 1455 raft_consensus.cc:368] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:12.098203 1455 raft_consensus.cc:722] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343052a5d259480dbf4599078770bfe1, State: Initialized, Role: FOLLOWER I0521 22:29:12.102499 1454 tablet_bootstrap.cc:596] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: No blocks or log segments found. Creating new log. I0521 22:29:12.104801 1455 consensus_queue.cc:250] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.107457 1460 raft_consensus.cc:478] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0521 22:29:12.108309 1460 raft_consensus.cc:500] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.124442 1454 tablet_bootstrap.cc:440] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: No bootstrap required, opened a new log I0521 22:29:12.124894 1454 ts_tablet_manager.cc:1115] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.012s I0521 22:29:12.125893 1455 ts_tablet_manager.cc:1144] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Time spent starting tablet: real 0.061s user 0.027s sys 0.018s I0521 22:29:12.127521 1455 tablet_bootstrap.cc:440] T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: Bootstrap starting. I0521 22:29:12.127521 1454 raft_consensus.cc:342] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.128301 1454 raft_consensus.cc:368] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:12.129695 1454 raft_consensus.cc:722] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4889a5698f944459719d537988dbaea, State: Initialized, Role: FOLLOWER I0521 22:29:12.130446 1454 consensus_queue.cc:250] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.138955 1454 ts_tablet_manager.cc:1144] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Time spent starting tablet: real 0.014s user 0.004s sys 0.002s I0521 22:29:12.113180 1460 leader_election.cc:288] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa747fc1bcba4e4fb55a9bce3a0c7c37 (127.0.89.194:46087), e4889a5698f944459719d537988dbaea (127.0.89.195:35691) I0521 22:29:12.154974 1264 tablet_service.cc:1504] Received RequestConsensusVote() RPC: tablet_id: "db62be61ab374802a1d543ea858f9113" candidate_uuid: "343052a5d259480dbf4599078770bfe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" is_pre_election: true W0521 22:29:12.165828 1061 leader_election.cc:341] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aa747fc1bcba4e4fb55a9bce3a0c7c37 (127.0.89.194:46087): Illegal state: must be running to vote when last-logged opid is not known I0521 22:29:12.166260 1455 tablet_bootstrap.cc:596] T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: No blocks or log segments found. Creating new log. I0521 22:29:11.935708 1243 tablet_service.cc:1232] Processing CreateTablet for tablet e7001908da8140df8082f3243384a351 (table=bench_manual_flush [id=9436caed107d45b6aac9e72ee79baf33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0521 22:29:12.176015 1243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7001908da8140df8082f3243384a351. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:12.178606 1455 tablet_bootstrap.cc:440] T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: No bootstrap required, opened a new log I0521 22:29:12.179059 1455 ts_tablet_manager.cc:1115] T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.019s I0521 22:29:12.179101 1390 tablet_service.cc:1504] Received RequestConsensusVote() RPC: tablet_id: "db62be61ab374802a1d543ea858f9113" candidate_uuid: "343052a5d259480dbf4599078770bfe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4889a5698f944459719d537988dbaea" is_pre_election: true I0521 22:29:12.179818 1390 raft_consensus.cc:2407] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 343052a5d259480dbf4599078770bfe1 in term 0. I0521 22:29:12.180881 1062 leader_election.cc:302] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 343052a5d259480dbf4599078770bfe1, e4889a5698f944459719d537988dbaea; no voters: aa747fc1bcba4e4fb55a9bce3a0c7c37 I0521 22:29:12.181468 1460 raft_consensus.cc:2721] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0521 22:29:12.181735 1460 raft_consensus.cc:478] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0521 22:29:12.182582 1460 raft_consensus.cc:2975] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:12.182999 1455 raft_consensus.cc:342] T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.183782 1455 raft_consensus.cc:368] T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:12.184067 1455 raft_consensus.cc:722] T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343052a5d259480dbf4599078770bfe1, State: Initialized, Role: FOLLOWER I0521 22:29:12.184885 1455 consensus_queue.cc:250] T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.187199 1456 tablet_bootstrap.cc:440] T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Bootstrap starting. I0521 22:29:12.187803 1455 ts_tablet_manager.cc:1144] T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s I0521 22:29:12.187628 1460 raft_consensus.cc:500] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.190265 1390 tablet_service.cc:1504] Received RequestConsensusVote() RPC: tablet_id: "db62be61ab374802a1d543ea858f9113" candidate_uuid: "343052a5d259480dbf4599078770bfe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4889a5698f944459719d537988dbaea" I0521 22:29:12.190110 1264 tablet_service.cc:1504] Received RequestConsensusVote() RPC: tablet_id: "db62be61ab374802a1d543ea858f9113" candidate_uuid: "343052a5d259480dbf4599078770bfe1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" I0521 22:29:12.190692 1390 raft_consensus.cc:2975] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Advancing to term 1 W0521 22:29:12.193406 1061 leader_election.cc:341] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer aa747fc1bcba4e4fb55a9bce3a0c7c37 (127.0.89.194:46087): Illegal state: must be running to vote when last-logged opid is not known I0521 22:29:12.196934 1460 leader_election.cc:288] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [CANDIDATE]: Term 1 election: Requested vote from peers aa747fc1bcba4e4fb55a9bce3a0c7c37 (127.0.89.194:46087), e4889a5698f944459719d537988dbaea (127.0.89.195:35691) I0521 22:29:12.199515 1390 raft_consensus.cc:2407] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 343052a5d259480dbf4599078770bfe1 in term 1. I0521 22:29:12.201617 1062 leader_election.cc:302] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 343052a5d259480dbf4599078770bfe1, e4889a5698f944459719d537988dbaea; no voters: aa747fc1bcba4e4fb55a9bce3a0c7c37 I0521 22:29:12.202075 1456 tablet_bootstrap.cc:596] T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: No blocks or log segments found. Creating new log. I0521 22:29:12.202605 1460 raft_consensus.cc:2721] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:12.203050 1460 raft_consensus.cc:683] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [term 1 LEADER]: Becoming Leader. State: Replica: 343052a5d259480dbf4599078770bfe1, State: Running, Role: LEADER I0521 22:29:12.204015 1456 log.cc:819] T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Log is configured to *not* fsync() on all Append() calls I0521 22:29:12.203876 1460 consensus_queue.cc:227] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.211246 1456 tablet_bootstrap.cc:440] T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: No bootstrap required, opened a new log I0521 22:29:12.211712 1456 ts_tablet_manager.cc:1115] T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.010s I0521 22:29:12.214108 1014 catalog_manager.cc:4202] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 reported cstate change: term changed from 0 to 1, leader changed from to 343052a5d259480dbf4599078770bfe1 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "343052a5d259480dbf4599078770bfe1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } health_report { overall_health: UNKNOWN } } } I0521 22:29:12.223186 1456 raft_consensus.cc:342] T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.223824 1456 raft_consensus.cc:368] T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:12.224018 1456 raft_consensus.cc:722] T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa747fc1bcba4e4fb55a9bce3a0c7c37, State: Initialized, Role: FOLLOWER W0521 22:29:12.224828 1182 tablet.cc:1820] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Can't schedule compaction. Clean time has not been advanced past its initial value. I0521 22:29:12.224735 1456 consensus_queue.cc:250] T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.228577 1456 ts_tablet_manager.cc:1144] T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Time spent starting tablet: real 0.017s user 0.018s sys 0.001s I0521 22:29:12.229214 1456 tablet_bootstrap.cc:440] T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Bootstrap starting. I0521 22:29:12.235227 1456 tablet_bootstrap.cc:596] T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: No blocks or log segments found. Creating new log. I0521 22:29:12.242450 1456 tablet_bootstrap.cc:440] T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: No bootstrap required, opened a new log I0521 22:29:12.242729 1456 ts_tablet_manager.cc:1115] T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s I0521 22:29:12.244442 1456 raft_consensus.cc:342] T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.244904 1456 raft_consensus.cc:368] T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:12.245062 1456 raft_consensus.cc:722] T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa747fc1bcba4e4fb55a9bce3a0c7c37, State: Initialized, Role: FOLLOWER I0521 22:29:12.245446 1456 consensus_queue.cc:250] T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.246686 1456 ts_tablet_manager.cc:1144] T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s W0521 22:29:12.296563 1434 tablet.cc:1820] T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Can't schedule compaction. Clean time has not been advanced past its initial value. I0521 22:29:12.326362 1459 raft_consensus.cc:478] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0521 22:29:12.326709 1459 raft_consensus.cc:500] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.328660 1459 leader_election.cc:288] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 343052a5d259480dbf4599078770bfe1 (127.0.89.193:45719), aa747fc1bcba4e4fb55a9bce3a0c7c37 (127.0.89.194:46087) I0521 22:29:12.344676 1138 tablet_service.cc:1504] Received RequestConsensusVote() RPC: tablet_id: "e7001908da8140df8082f3243384a351" candidate_uuid: "e4889a5698f944459719d537988dbaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343052a5d259480dbf4599078770bfe1" is_pre_election: true I0521 22:29:12.344980 1264 tablet_service.cc:1504] Received RequestConsensusVote() RPC: tablet_id: "e7001908da8140df8082f3243384a351" candidate_uuid: "e4889a5698f944459719d537988dbaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" is_pre_election: true I0521 22:29:12.345383 1138 raft_consensus.cc:2407] T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4889a5698f944459719d537988dbaea in term 0. I0521 22:29:12.345527 1264 raft_consensus.cc:2407] T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4889a5698f944459719d537988dbaea in term 0. I0521 22:29:12.347998 1320 leader_election.cc:302] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa747fc1bcba4e4fb55a9bce3a0c7c37, e4889a5698f944459719d537988dbaea; no voters: I0521 22:29:12.348734 1459 raft_consensus.cc:2721] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Leader pre-election won for term 1 I0521 22:29:12.348976 1459 raft_consensus.cc:478] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0521 22:29:12.349187 1459 raft_consensus.cc:2975] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:12.352537 1459 raft_consensus.cc:500] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } I0521 22:29:12.353763 1459 leader_election.cc:288] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [CANDIDATE]: Term 1 election: Requested vote from peers 343052a5d259480dbf4599078770bfe1 (127.0.89.193:45719), aa747fc1bcba4e4fb55a9bce3a0c7c37 (127.0.89.194:46087) I0521 22:29:12.354454 1138 tablet_service.cc:1504] Received RequestConsensusVote() RPC: tablet_id: "e7001908da8140df8082f3243384a351" candidate_uuid: "e4889a5698f944459719d537988dbaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343052a5d259480dbf4599078770bfe1" I0521 22:29:12.354640 1264 tablet_service.cc:1504] Received RequestConsensusVote() RPC: tablet_id: "e7001908da8140df8082f3243384a351" candidate_uuid: "e4889a5698f944459719d537988dbaea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" I0521 22:29:12.354809 1138 raft_consensus.cc:2975] T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:12.356305 1264 raft_consensus.cc:2975] T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:12.360388 1138 raft_consensus.cc:2407] T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4889a5698f944459719d537988dbaea in term 1. I0521 22:29:12.360435 1264 raft_consensus.cc:2407] T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4889a5698f944459719d537988dbaea in term 1. I0521 22:29:12.361196 1321 leader_election.cc:302] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343052a5d259480dbf4599078770bfe1, e4889a5698f944459719d537988dbaea; no voters: I0521 22:29:12.361737 1459 raft_consensus.cc:2721] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:12.362851 1459 raft_consensus.cc:683] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [term 1 LEADER]: Becoming Leader. State: Replica: e4889a5698f944459719d537988dbaea, State: Running, Role: LEADER I0521 22:29:12.364709 1459 consensus_queue.cc:227] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } } W0521 22:29:12.370155 1308 tablet.cc:1820] T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Can't schedule compaction. Clean time has not been advanced past its initial value. I0521 22:29:12.371588 1014 catalog_manager.cc:4202] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea reported cstate change: term changed from 0 to 1, leader changed from to e4889a5698f944459719d537988dbaea (127.0.89.195). New cstate: current_term: 1 leader_uuid: "e4889a5698f944459719d537988dbaea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 } health_report { overall_health: HEALTHY } } } I0521 22:29:12.607981 1460 consensus_queue.cc:1003] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4889a5698f944459719d537988dbaea" member_type: VOTER last_known_addr { host: "127.0.89.195" port: 35691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0521 22:29:12.627355 1264 raft_consensus.cc:2975] T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:12.640710 1461 consensus_queue.cc:1003] T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0521 22:29:12.662631 1474 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:29:12.663349 1474 flags.cc:403] Enabled unsafe flag: --never_fsync=true I0521 22:29:12.895987 1459 consensus_queue.cc:1003] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "343052a5d259480dbf4599078770bfe1" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0521 22:29:12.913938 1459 consensus_queue.cc:1003] T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa747fc1bcba4e4fb55a9bce3a0c7c37" member_type: VOTER last_known_addr { host: "127.0.89.194" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0521 22:29:13.832960 1474 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 1.113s user 0.002s sys 0.000s W0521 22:29:13.833196 1474 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 1.113s user 0.002s sys 0.000s Using table 'bench_manual_flush' W0521 22:29:14.612097 1507 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.091s user 0.011s sys 0.012s W0521 22:29:14.675644 1512 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.093s user 0.019s sys 0.000s W0521 22:29:14.865909 1515 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.113s user 0.025s sys 0.000s W0521 22:29:14.882180 1512 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.051s user 0.021s sys 0.000s W0521 22:29:14.883325 1507 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.116s user 0.020s sys 0.002s W0521 22:29:14.893787 1510 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.089s user 0.019s sys 0.005s W0521 22:29:14.960158 1518 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.166s user 0.015s sys 0.011s W0521 22:29:15.024173 1519 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.183s user 0.018s sys 0.006s W0521 22:29:15.115633 1515 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.061s user 0.021s sys 0.003s W0521 22:29:15.198767 1519 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.111s user 0.020s sys 0.001s W0521 22:29:15.223415 1518 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.113s user 0.023s sys 0.000s W0521 22:29:15.368443 1512 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.063s user 0.023s sys 0.000s W0521 22:29:15.421121 1507 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.140s user 0.022s sys 0.002s W0521 22:29:15.480876 1510 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.169s user 0.023s sys 0.000s W0521 22:29:15.491575 1515 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.112s user 0.009s sys 0.011s W0521 22:29:15.515008 1519 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.078s user 0.019s sys 0.000s W0521 22:29:15.538266 1518 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.103s user 0.016s sys 0.004s I0521 22:29:15.826570 1525 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47006 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=1, attempt_no=0}) took 1452ms. Trace: I0521 22:29:15.827020 1525 rpcz_store.cc:270] 0521 22:29:14.374182 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:14.374310 (+ 128us) service_pool.cc:223] Handling call 0521 22:29:15.826410 (+1452100us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:14.408520 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:14.409385 (+ 865us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:14.409412 (+ 27us) write_op.cc:338] Acquired schema lock 0521 22:29:14.409426 (+ 14us) tablet.cc:452] Decoding operations 0521 22:29:14.549092 (+139666us) tablet.cc:474] Acquiring locks for 533 operations 0521 22:29:14.745197 (+196105us) tablet.cc:479] Locks acquired 0521 22:29:14.745214 (+ 17us) write_op.cc:186] PREPARE: Finished. 0521 22:29:14.745454 (+ 240us) write_op.cc:196] Start() 0521 22:29:14.745565 (+ 111us) write_op.cc:201] Timestamp: P: 1590100154745392 usec, L: 0 0521 22:29:14.745593 (+ 28us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:14.747421 (+ 1828us) log.cc:840] Serialized 74645 byte log entry 0521 22:29:14.966118 (+218697us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:14.980718 (+ 14600us) write_op.cc:225] APPLY: Starting. 0521 22:29:15.614413 (+633695us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b5c0004fff0. stack: 00007f5d60b8cac3 00007f5d60b8c9c0 00007f5d6042da14 00007f5d69e5c154 00007f5d69e5c0e9 00007f5d68c02c81 00007f5d68be9b75 00007f5d68be992a 00007f5d68ca5a05 00007f5d68c94bf9 00007f5d68c995e1 00007f5d68c99571 00007f5d68c99501 00007f5d68c994c1 00007f5d68c9855d 00007f5d6404c0f4 0521 22:29:15.736733 (+122320us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:15.736776 (+ 43us) write_op.cc:236] APPLY: Finished. 0521 22:29:15.757305 (+ 20529us) log.cc:840] Serialized 3217 byte log entry 0521 22:29:15.773640 (+ 16335us) write_op.cc:379] Releasing row and schema locks 0521 22:29:15.816834 (+ 43194us) write_op.cc:344] Released schema lock 0521 22:29:15.826092 (+ 9258us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":14433,"num_ops":533,"prepare.queue_time_us":5075,"prepare.run_cpu_time_us":99784,"prepare.run_wall_time_us":368395,"raft.queue_time_us":223,"raft.run_cpu_time_us":4590,"raft.run_wall_time_us":10659,"replication_time_us":220426,"spinlock_wait_cycles":31952512,"thread_start_us":27270,"threads_started":3,"wal-append.queue_time_us":4948}]]} W0521 22:29:15.899062 1488 outbound_call.cc:285] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.89.195:35691, user_credentials={real_user=slave}} blocked reactor thread for 65416.6us W0521 22:29:15.961350 1510 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.088s user 0.023s sys 0.000s I0521 22:29:16.012696 1522 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36428 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=2, attempt_no=0}) took 1586ms. Trace: I0521 22:29:16.013098 1522 rpcz_store.cc:270] 0521 22:29:14.425877 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:14.426022 (+ 145us) service_pool.cc:223] Handling call 0521 22:29:16.012540 (+1586518us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:14.552175 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:14.553060 (+ 885us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:14.553085 (+ 25us) write_op.cc:338] Acquired schema lock 0521 22:29:14.553096 (+ 11us) tablet.cc:452] Decoding operations 0521 22:29:14.650263 (+ 97167us) tablet.cc:474] Acquiring locks for 514 operations 0521 22:29:14.765784 (+115521us) tablet.cc:479] Locks acquired 0521 22:29:14.765800 (+ 16us) write_op.cc:186] PREPARE: Finished. 0521 22:29:14.765987 (+ 187us) write_op.cc:196] Start() 0521 22:29:14.766075 (+ 88us) write_op.cc:201] Timestamp: P: 1590100154765942 usec, L: 0 0521 22:29:14.766092 (+ 17us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:14.767456 (+ 1364us) log.cc:840] Serialized 72004 byte log entry 0521 22:29:14.902237 (+134781us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:14.925325 (+ 23088us) write_op.cc:225] APPLY: Starting. 0521 22:29:15.825149 (+899824us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:15.825191 (+ 42us) write_op.cc:236] APPLY: Finished. 0521 22:29:15.897087 (+ 71896us) log.cc:840] Serialized 3103 byte log entry 0521 22:29:15.897208 (+ 121us) write_op.cc:379] Releasing row and schema locks 0521 22:29:15.975091 (+ 77883us) write_op.cc:344] Released schema lock 0521 22:29:16.012309 (+ 37218us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":22911,"mutex_wait_us":8059,"num_ops":514,"prepare.queue_time_us":122824,"prepare.run_cpu_time_us":76439,"prepare.run_wall_time_us":269840,"raft.queue_time_us":395,"raft.run_cpu_time_us":3490,"raft.run_wall_time_us":3633,"replication_time_us":136064,"spinlock_wait_cycles":403584,"thread_start_us":21601,"threads_started":1}]]} W0521 22:29:16.034497 1507 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.204s user 0.010s sys 0.013s I0521 22:29:16.042179 1527 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47006 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=3, attempt_no=0}) took 1528ms. Trace: W0521 22:29:16.064389 1519 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.123s user 0.022s sys 0.000s I0521 22:29:16.054400 1527 rpcz_store.cc:270] 0521 22:29:14.512964 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:14.513132 (+ 168us) service_pool.cc:223] Handling call 0521 22:29:16.040913 (+1527781us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:14.777007 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:14.777821 (+ 814us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:14.777842 (+ 21us) write_op.cc:338] Acquired schema lock 0521 22:29:14.777855 (+ 13us) tablet.cc:452] Decoding operations 0521 22:29:14.912676 (+134821us) tablet.cc:474] Acquiring locks for 511 operations 0521 22:29:15.052303 (+139627us) tablet.cc:479] Locks acquired 0521 22:29:15.052331 (+ 28us) write_op.cc:186] PREPARE: Finished. 0521 22:29:15.052530 (+ 199us) write_op.cc:196] Start() 0521 22:29:15.052633 (+ 103us) write_op.cc:201] Timestamp: P: 1590100155052481 usec, L: 0 0521 22:29:15.052652 (+ 19us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:15.054459 (+ 1807us) log.cc:840] Serialized 71586 byte log entry 0521 22:29:15.215154 (+160695us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:15.239301 (+ 24147us) write_op.cc:225] APPLY: Starting. 0521 22:29:15.440752 (+201451us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b5c0004fff0. stack: 00007f5d60b8cac3 00007f5d60b8c9c0 00007f5d6042da14 00007f5d69e5c154 00007f5d69e5c0e9 00007f5d68c02c81 00007f5d68be9b75 00007f5d68be992a 00007f5d68ca5a05 00007f5d68c94bf9 00007f5d68c995e1 00007f5d68c99571 00007f5d68c99501 00007f5d68c994c1 00007f5d68c9855d 00007f5d6404c0f4 0521 22:29:15.880421 (+439669us) spinlock_profiling.cc:243] Waited 8.25 ms on lock 0x7b5c0004fff0. stack: 00007f5d60b8cac3 00007f5d60b8c9c0 00007f5d6042da14 00007f5d69e5c154 00007f5d69e5c0e9 00007f5d68c02c81 00007f5d68be9b75 00007f5d68be992a 00007f5d68ca5a05 00007f5d68c94bf9 00007f5d68c995e1 00007f5d68c99571 00007f5d68c99501 00007f5d68c994c1 00007f5d68c9855d 00007f5d6404c0f4 0521 22:29:16.002052 (+121631us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:16.002093 (+ 41us) write_op.cc:236] APPLY: Finished. 0521 22:29:16.025910 (+ 23817us) log.cc:840] Serialized 3085 byte log entry 0521 22:29:16.026034 (+ 124us) write_op.cc:379] Releasing row and schema locks 0521 22:29:16.037399 (+ 11365us) write_op.cc:344] Released schema lock 0521 22:29:16.040674 (+ 3275us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":23956,"mutex_wait_us":15154,"num_ops":511,"prepare.queue_time_us":260259,"prepare.run_cpu_time_us":83806,"prepare.run_wall_time_us":299178,"raft.queue_time_us":9266,"raft.run_cpu_time_us":3471,"raft.run_wall_time_us":8116,"replication_time_us":162419,"spinlock_wait_cycles":27667200,"thread_start_us":24747,"threads_started":1}]]} W0521 22:29:16.081437 1515 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.175s user 0.021s sys 0.003s W0521 22:29:16.091084 1512 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.190s user 0.022s sys 0.000s I0521 22:29:16.230126 1521 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47006 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=5, attempt_no=0}) took 1510ms. Trace: I0521 22:29:16.230433 1521 rpcz_store.cc:270] 0521 22:29:14.720250 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:14.720425 (+ 175us) service_pool.cc:223] Handling call 0521 22:29:16.230060 (+1509635us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:15.076303 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:15.077118 (+ 815us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:15.077139 (+ 21us) write_op.cc:338] Acquired schema lock 0521 22:29:15.077151 (+ 12us) tablet.cc:452] Decoding operations 0521 22:29:15.165954 (+ 88803us) tablet.cc:474] Acquiring locks for 500 operations 0521 22:29:15.376720 (+210766us) tablet.cc:479] Locks acquired 0521 22:29:15.376736 (+ 16us) write_op.cc:186] PREPARE: Finished. 0521 22:29:15.376940 (+ 204us) write_op.cc:196] Start() 0521 22:29:15.377044 (+ 104us) write_op.cc:201] Timestamp: P: 1590100155376890 usec, L: 0 0521 22:29:15.377060 (+ 16us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:15.379684 (+ 2624us) log.cc:840] Serialized 70057 byte log entry 0521 22:29:15.526158 (+146474us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:15.526395 (+ 237us) write_op.cc:225] APPLY: Starting. 0521 22:29:15.612543 (+ 86148us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b5c0004fff0. stack: 00007f5d60b8cac3 00007f5d60b8c9c0 00007f5d6042da14 00007f5d69e5c154 00007f5d69e5c0e9 00007f5d68c02c81 00007f5d68be9b75 00007f5d68be992a 00007f5d68ca5a05 00007f5d68c94bf9 00007f5d68c995e1 00007f5d68c99571 00007f5d68c99501 00007f5d68c994c1 00007f5d68c9855d 00007f5d6404c0f4 0521 22:29:16.175810 (+563267us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:16.175853 (+ 43us) write_op.cc:236] APPLY: Finished. 0521 22:29:16.191321 (+ 15468us) log.cc:840] Serialized 3019 byte log entry 0521 22:29:16.196137 (+ 4816us) write_op.cc:379] Releasing row and schema locks 0521 22:29:16.215121 (+ 18984us) write_op.cc:344] Released schema lock 0521 22:29:16.229842 (+ 14721us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":102,"num_ops":500,"prepare.queue_time_us":351992,"prepare.run_cpu_time_us":77110,"prepare.run_wall_time_us":347469,"raft.queue_time_us":237,"raft.run_cpu_time_us":3368,"raft.run_wall_time_us":8519,"replication_time_us":149009,"spinlock_wait_cycles":28019712}]]} W0521 22:29:16.382863 1515 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.079s user 0.013s sys 0.005s W0521 22:29:16.415535 1519 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.079s user 0.011s sys 0.008s I0521 22:29:16.419435 1513 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36428 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=4, attempt_no=0}) took 1807ms. Trace: I0521 22:29:16.428115 1513 rpcz_store.cc:270] 0521 22:29:14.611860 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:14.612004 (+ 144us) service_pool.cc:223] Handling call 0521 22:29:16.419370 (+1807366us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:14.621984 (+ 0us) spinlock_profiling.cc:243] Waited 5.11 ms on lock 0x7b50000812b4. stack: 00007f3359a47ac3 00007f3359a479c0 00007f33592e8a14 00007f3362d17154 00007f3362d170e9 00007f33615f1351 00007f3361666fda 00007f3361b4d703 00007f3361b2dccc 00007f3362d991ce 00007f335c79d314 00007f335c79d26d 00007f335c79d18c 00007f335c79d0fc 00007f335c79c152 00007f335b88c99c 0521 22:29:14.822154 (+200170us) write_op.cc:151] PREPARE: Starting. 0521 22:29:14.823011 (+ 857us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:14.823042 (+ 31us) write_op.cc:338] Acquired schema lock 0521 22:29:14.823059 (+ 17us) tablet.cc:452] Decoding operations 0521 22:29:14.970577 (+147518us) tablet.cc:474] Acquiring locks for 525 operations 0521 22:29:15.274544 (+303967us) tablet.cc:479] Locks acquired 0521 22:29:15.274564 (+ 20us) write_op.cc:186] PREPARE: Finished. 0521 22:29:15.274764 (+ 200us) write_op.cc:196] Start() 0521 22:29:15.274864 (+ 100us) write_op.cc:201] Timestamp: P: 1590100155274712 usec, L: 0 0521 22:29:15.274882 (+ 18us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:15.280516 (+ 5634us) log.cc:840] Serialized 73533 byte log entry 0521 22:29:15.423752 (+143236us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:15.424245 (+ 493us) write_op.cc:225] APPLY: Starting. 0521 22:29:16.310257 (+886012us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:16.310300 (+ 43us) write_op.cc:236] APPLY: Finished. 0521 22:29:16.341725 (+ 31425us) log.cc:840] Serialized 3169 byte log entry 0521 22:29:16.343668 (+ 1943us) write_op.cc:379] Releasing row and schema locks 0521 22:29:16.383888 (+ 40220us) write_op.cc:344] Released schema lock 0521 22:29:16.419143 (+ 35255us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":300,"num_ops":525,"prepare.queue_time_us":200082,"prepare.run_cpu_time_us":78984,"prepare.run_wall_time_us":489942,"raft.queue_time_us":10758,"raft.run_cpu_time_us":4710,"raft.run_wall_time_us":5574,"replication_time_us":148786,"spinlock_wait_cycles":12315264}]]} W0521 22:29:16.507653 1518 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.144s user 0.010s sys 0.010s W0521 22:29:16.689369 1512 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.053s user 0.021s sys 0.000s W0521 22:29:16.757261 1507 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.171s user 0.022s sys 0.000s W0521 22:29:16.803462 1510 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.204s user 0.023s sys 0.000s I0521 22:29:16.852072 1527 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47006 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=6, attempt_no=0}) took 2087ms. Trace: I0521 22:29:16.852458 1527 rpcz_store.cc:270] 0521 22:29:14.758556 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:14.760112 (+ 1556us) service_pool.cc:223] Handling call 0521 22:29:16.845954 (+2085842us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:15.423898 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:15.427513 (+ 3615us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:15.427535 (+ 22us) write_op.cc:338] Acquired schema lock 0521 22:29:15.427548 (+ 13us) tablet.cc:452] Decoding operations 0521 22:29:15.658873 (+231325us) tablet.cc:474] Acquiring locks for 522 operations 0521 22:29:15.903806 (+244933us) tablet.cc:479] Locks acquired 0521 22:29:15.903824 (+ 18us) write_op.cc:186] PREPARE: Finished. 0521 22:29:15.904012 (+ 188us) write_op.cc:196] Start() 0521 22:29:15.904145 (+ 133us) write_op.cc:201] Timestamp: P: 1590100155903961 usec, L: 0 0521 22:29:15.904162 (+ 17us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:15.905858 (+ 1696us) log.cc:840] Serialized 73116 byte log entry 0521 22:29:16.104263 (+198405us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:16.105032 (+ 769us) write_op.cc:225] APPLY: Starting. 0521 22:29:16.789588 (+684556us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:16.789639 (+ 51us) write_op.cc:236] APPLY: Finished. 0521 22:29:16.797644 (+ 8005us) log.cc:840] Serialized 3151 byte log entry 0521 22:29:16.799839 (+ 2195us) write_op.cc:379] Releasing row and schema locks 0521 22:29:16.842726 (+ 42887us) write_op.cc:344] Released schema lock 0521 22:29:16.845717 (+ 2991us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":641,"mutex_wait_us":65932,"num_ops":522,"prepare.queue_time_us":660280,"prepare.run_cpu_time_us":78168,"prepare.run_wall_time_us":501253,"raft.queue_time_us":224,"raft.run_cpu_time_us":3761,"raft.run_wall_time_us":7286,"replication_time_us":200001,"spinlock_wait_cycles":612352}]]} I0521 22:29:16.958429 1521 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47006 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=9, attempt_no=0}) took 1381ms. Trace: I0521 22:29:16.958827 1521 rpcz_store.cc:270] 0521 22:29:15.577131 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:15.577273 (+ 142us) service_pool.cc:223] Handling call 0521 22:29:16.958362 (+1381089us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:15.925264 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:15.926031 (+ 767us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:15.926051 (+ 20us) write_op.cc:338] Acquired schema lock 0521 22:29:15.926063 (+ 12us) tablet.cc:452] Decoding operations 0521 22:29:16.170359 (+244296us) tablet.cc:474] Acquiring locks for 483 operations 0521 22:29:16.302103 (+131744us) tablet.cc:479] Locks acquired 0521 22:29:16.302123 (+ 20us) write_op.cc:186] PREPARE: Finished. 0521 22:29:16.302313 (+ 190us) write_op.cc:196] Start() 0521 22:29:16.302411 (+ 98us) write_op.cc:201] Timestamp: P: 1590100156302265 usec, L: 0 0521 22:29:16.302428 (+ 17us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:16.303825 (+ 1397us) log.cc:840] Serialized 67703 byte log entry 0521 22:29:16.436774 (+132949us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:16.437095 (+ 321us) write_op.cc:225] APPLY: Starting. 0521 22:29:16.946825 (+509730us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:16.946866 (+ 41us) write_op.cc:236] APPLY: Finished. 0521 22:29:16.950162 (+ 3296us) log.cc:840] Serialized 2917 byte log entry 0521 22:29:16.950275 (+ 113us) write_op.cc:379] Releasing row and schema locks 0521 22:29:16.955367 (+ 5092us) write_op.cc:344] Released schema lock 0521 22:29:16.958130 (+ 2763us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":189,"num_ops":483,"prepare.queue_time_us":334850,"prepare.run_cpu_time_us":68877,"prepare.run_wall_time_us":426002,"raft.queue_time_us":252,"raft.run_cpu_time_us":3897,"raft.run_wall_time_us":4573,"replication_time_us":134261,"spinlock_wait_cycles":577536}]]} W0521 22:29:16.965538 1519 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.065s user 0.021s sys 0.000s W0521 22:29:16.970722 1515 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.097s user 0.016s sys 0.004s W0521 22:29:16.975991 1518 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.068s user 0.019s sys 0.000s I0521 22:29:17.197940 1522 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36428 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=7, attempt_no=0}) took 2381ms. Trace: I0521 22:29:17.198228 1522 rpcz_store.cc:270] 0521 22:29:14.816158 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:14.816300 (+ 142us) service_pool.cc:223] Handling call 0521 22:29:17.197875 (+2381575us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:15.312238 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:15.320182 (+ 7944us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:15.320204 (+ 22us) write_op.cc:338] Acquired schema lock 0521 22:29:15.320216 (+ 12us) tablet.cc:452] Decoding operations 0521 22:29:15.568977 (+248761us) tablet.cc:474] Acquiring locks for 502 operations 0521 22:29:15.828522 (+259545us) tablet.cc:479] Locks acquired 0521 22:29:15.828537 (+ 15us) write_op.cc:186] PREPARE: Finished. 0521 22:29:15.828725 (+ 188us) write_op.cc:196] Start() 0521 22:29:15.828831 (+ 106us) write_op.cc:201] Timestamp: P: 1590100155828680 usec, L: 0 0521 22:29:15.828848 (+ 17us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:15.830205 (+ 1357us) log.cc:840] Serialized 70335 byte log entry 0521 22:29:16.066325 (+236120us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:16.085889 (+ 19564us) write_op.cc:225] APPLY: Starting. 0521 22:29:17.047918 (+962029us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:17.047961 (+ 43us) write_op.cc:236] APPLY: Finished. 0521 22:29:17.083471 (+ 35510us) log.cc:840] Serialized 3031 byte log entry 0521 22:29:17.085470 (+ 1999us) write_op.cc:379] Releasing row and schema locks 0521 22:29:17.194860 (+109390us) write_op.cc:344] Released schema lock 0521 22:29:17.197636 (+ 2776us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":19300,"num_ops":502,"prepare.queue_time_us":491909,"prepare.run_cpu_time_us":71815,"prepare.run_wall_time_us":578436,"raft.queue_time_us":260,"raft.run_cpu_time_us":3364,"raft.run_wall_time_us":4015,"replication_time_us":237395,"spinlock_wait_cycles":659456}]]} W0521 22:29:17.294448 1519 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.066s user 0.017s sys 0.000s W0521 22:29:17.296202 1515 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.098s user 0.017s sys 0.002s I0521 22:29:17.377728 1521 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47006 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=10, attempt_no=0}) took 1723ms. Trace: I0521 22:29:17.378415 1521 rpcz_store.cc:270] 0521 22:29:15.653981 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:15.654172 (+ 191us) service_pool.cc:223] Handling call 0521 22:29:17.377573 (+1723401us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:16.351384 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:16.356227 (+ 4843us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:16.356250 (+ 23us) write_op.cc:338] Acquired schema lock 0521 22:29:16.356262 (+ 12us) tablet.cc:452] Decoding operations 0521 22:29:16.530744 (+174482us) tablet.cc:474] Acquiring locks for 518 operations 0521 22:29:16.860019 (+329275us) tablet.cc:479] Locks acquired 0521 22:29:16.871132 (+ 11113us) write_op.cc:186] PREPARE: Finished. 0521 22:29:16.871314 (+ 182us) write_op.cc:196] Start() 0521 22:29:16.871407 (+ 93us) write_op.cc:201] Timestamp: P: 1590100156871263 usec, L: 0 0521 22:29:16.871423 (+ 16us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:16.873275 (+ 1852us) log.cc:840] Serialized 72569 byte log entry 0521 22:29:16.985068 (+111793us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:16.991826 (+ 6758us) write_op.cc:225] APPLY: Starting. 0521 22:29:17.344982 (+353156us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:17.345026 (+ 44us) write_op.cc:236] APPLY: Finished. 0521 22:29:17.356963 (+ 11937us) log.cc:840] Serialized 3127 byte log entry 0521 22:29:17.357107 (+ 144us) write_op.cc:379] Releasing row and schema locks 0521 22:29:17.371416 (+ 14309us) write_op.cc:344] Released schema lock 0521 22:29:17.377161 (+ 5745us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":6607,"num_ops":518,"prepare.queue_time_us":693748,"prepare.run_cpu_time_us":81659,"prepare.run_wall_time_us":571360,"raft.queue_time_us":4574,"raft.run_cpu_time_us":3236,"raft.run_wall_time_us":14133,"replication_time_us":113571,"spinlock_wait_cycles":570112}]]} W0521 22:29:17.403391 1510 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.098s user 0.016s sys 0.005s W0521 22:29:17.404350 1507 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.076s user 0.021s sys 0.000s W0521 22:29:17.413228 1518 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.136s user 0.021s sys 0.000s W0521 22:29:17.476694 1512 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.116s user 0.018s sys 0.007s I0521 22:29:17.833900 1527 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47006 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=12, attempt_no=0}) took 1022ms. Trace: I0521 22:29:17.836115 1527 rpcz_store.cc:270] 0521 22:29:16.811523 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:16.811711 (+ 188us) service_pool.cc:223] Handling call 0521 22:29:17.833834 (+1022123us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:16.922884 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:16.923695 (+ 811us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:16.923717 (+ 22us) write_op.cc:338] Acquired schema lock 0521 22:29:16.923728 (+ 11us) tablet.cc:452] Decoding operations 0521 22:29:17.054730 (+131002us) tablet.cc:474] Acquiring locks for 499 operations 0521 22:29:17.192313 (+137583us) tablet.cc:479] Locks acquired 0521 22:29:17.192332 (+ 19us) write_op.cc:186] PREPARE: Finished. 0521 22:29:17.192522 (+ 190us) write_op.cc:196] Start() 0521 22:29:17.192618 (+ 96us) write_op.cc:201] Timestamp: P: 1590100157192479 usec, L: 0 0521 22:29:17.192636 (+ 18us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:17.197953 (+ 5317us) log.cc:840] Serialized 69927 byte log entry 0521 22:29:17.309326 (+111373us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:17.309657 (+ 331us) write_op.cc:225] APPLY: Starting. 0521 22:29:17.797914 (+488257us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:17.797958 (+ 44us) write_op.cc:236] APPLY: Finished. 0521 22:29:17.811321 (+ 13363us) log.cc:840] Serialized 3013 byte log entry 0521 22:29:17.816140 (+ 4819us) write_op.cc:379] Releasing row and schema locks 0521 22:29:17.826662 (+ 10522us) write_op.cc:344] Released schema lock 0521 22:29:17.833598 (+ 6936us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":201,"num_ops":499,"prepare.queue_time_us":97773,"prepare.run_cpu_time_us":69597,"prepare.run_wall_time_us":325453,"raft.queue_time_us":4187,"raft.run_cpu_time_us":3629,"raft.run_wall_time_us":9029,"replication_time_us":116605,"spinlock_wait_cycles":652672}]]} I0521 22:29:17.951586 1531 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36428 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=8, attempt_no=0}) took 2499ms. Trace: I0521 22:29:17.952009 1531 rpcz_store.cc:270] 0521 22:29:15.452591 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:15.452746 (+ 155us) service_pool.cc:223] Handling call 0521 22:29:17.951385 (+2498639us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:15.890799 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:15.891594 (+ 795us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:15.891616 (+ 22us) write_op.cc:338] Acquired schema lock 0521 22:29:15.891628 (+ 12us) tablet.cc:452] Decoding operations 0521 22:29:16.202267 (+310639us) tablet.cc:474] Acquiring locks for 541 operations 0521 22:29:16.531403 (+329136us) tablet.cc:479] Locks acquired 0521 22:29:16.531420 (+ 17us) write_op.cc:186] PREPARE: Finished. 0521 22:29:16.531649 (+ 229us) write_op.cc:196] Start() 0521 22:29:16.531756 (+ 107us) write_op.cc:201] Timestamp: P: 1590100156531596 usec, L: 0 0521 22:29:16.531774 (+ 18us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:16.585751 (+ 53977us) log.cc:840] Serialized 75766 byte log entry 0521 22:29:16.806856 (+221105us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:16.946920 (+140064us) write_op.cc:225] APPLY: Starting. 0521 22:29:17.835394 (+888474us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:17.835453 (+ 59us) write_op.cc:236] APPLY: Finished. 0521 22:29:17.865079 (+ 29626us) log.cc:840] Serialized 3265 byte log entry 0521 22:29:17.867344 (+ 2265us) write_op.cc:379] Releasing row and schema locks 0521 22:29:17.886189 (+ 18845us) write_op.cc:344] Released schema lock 0521 22:29:17.951077 (+ 64888us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":139933,"num_ops":541,"prepare.queue_time_us":434353,"prepare.run_cpu_time_us":74959,"prepare.run_wall_time_us":733655,"raft.queue_time_us":291,"raft.run_cpu_time_us":3975,"raft.run_wall_time_us":21576,"replication_time_us":275001,"spinlock_wait_cycles":687488}]]} W0521 22:29:17.998811 1507 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.100s user 0.018s sys 0.002s W0521 22:29:18.023237 1512 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.089s user 0.019s sys 0.000s W0521 22:29:18.019887 1510 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.079s user 0.019s sys 0.000s I0521 22:29:18.104651 1536 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36428 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=11, attempt_no=0}) took 2355ms. Trace: I0521 22:29:18.105445 1536 rpcz_store.cc:270] 0521 22:29:15.749489 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:15.749632 (+ 143us) service_pool.cc:223] Handling call 0521 22:29:18.104494 (+2354862us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:16.624560 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:16.625315 (+ 755us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:16.625347 (+ 32us) write_op.cc:338] Acquired schema lock 0521 22:29:16.625360 (+ 13us) tablet.cc:452] Decoding operations 0521 22:29:16.939018 (+313658us) tablet.cc:474] Acquiring locks for 506 operations 0521 22:29:17.283030 (+344012us) tablet.cc:479] Locks acquired 0521 22:29:17.283046 (+ 16us) write_op.cc:186] PREPARE: Finished. 0521 22:29:17.283218 (+ 172us) write_op.cc:196] Start() 0521 22:29:17.283305 (+ 87us) write_op.cc:201] Timestamp: P: 1590100157283176 usec, L: 0 0521 22:29:17.283333 (+ 28us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:17.328495 (+ 45162us) log.cc:840] Serialized 70900 byte log entry 0521 22:29:17.425359 (+ 96864us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:17.440192 (+ 14833us) write_op.cc:225] APPLY: Starting. 0521 22:29:18.045402 (+605210us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:18.047926 (+ 2524us) write_op.cc:236] APPLY: Finished. 0521 22:29:18.068242 (+ 20316us) log.cc:840] Serialized 3055 byte log entry 0521 22:29:18.070764 (+ 2522us) write_op.cc:379] Releasing row and schema locks 0521 22:29:18.097733 (+ 26969us) write_op.cc:344] Released schema lock 0521 22:29:18.104248 (+ 6515us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":14697,"num_ops":506,"prepare.queue_time_us":872662,"prepare.run_cpu_time_us":91973,"prepare.run_wall_time_us":704212,"raft.queue_time_us":3464,"raft.run_cpu_time_us":2188,"raft.run_wall_time_us":16182,"replication_time_us":141923,"spinlock_wait_cycles":27392}]]} I0521 22:29:18.240784 1531 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36428 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=13, attempt_no=0}) took 1321ms. Trace: I0521 22:29:18.241212 1531 rpcz_store.cc:270] 0521 22:29:16.919857 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:16.920050 (+ 193us) service_pool.cc:223] Handling call 0521 22:29:18.240683 (+1320633us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:17.328885 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:17.329664 (+ 779us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:17.329685 (+ 21us) write_op.cc:338] Acquired schema lock 0521 22:29:17.329697 (+ 12us) tablet.cc:452] Decoding operations 0521 22:29:17.620315 (+290618us) tablet.cc:474] Acquiring locks for 525 operations 0521 22:29:17.883168 (+262853us) tablet.cc:479] Locks acquired 0521 22:29:17.883190 (+ 22us) write_op.cc:186] PREPARE: Finished. 0521 22:29:17.883436 (+ 246us) write_op.cc:196] Start() 0521 22:29:17.883537 (+ 101us) write_op.cc:201] Timestamp: P: 1590100157883390 usec, L: 0 0521 22:29:17.883554 (+ 17us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:17.898822 (+ 15268us) log.cc:840] Serialized 73542 byte log entry 0521 22:29:18.070974 (+172152us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:18.071281 (+ 307us) write_op.cc:225] APPLY: Starting. 0521 22:29:18.225247 (+153966us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:18.225290 (+ 43us) write_op.cc:236] APPLY: Finished. 0521 22:29:18.228912 (+ 3622us) log.cc:840] Serialized 3169 byte log entry 0521 22:29:18.231045 (+ 2133us) write_op.cc:379] Releasing row and schema locks 0521 22:29:18.237396 (+ 6351us) write_op.cc:344] Released schema lock 0521 22:29:18.240397 (+ 3001us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":125,"num_ops":525,"prepare.queue_time_us":406318,"prepare.run_cpu_time_us":89080,"prepare.run_wall_time_us":592860,"raft.queue_time_us":469,"raft.run_cpu_time_us":3408,"raft.run_wall_time_us":5425,"replication_time_us":187319,"spinlock_wait_cycles":282880}]]} W0521 22:29:18.347138 1515 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.092s user 0.018s sys 0.000s W0521 22:29:18.351174 1519 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.076s user 0.019s sys 0.000s W0521 22:29:18.504309 1510 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.077s user 0.014s sys 0.009s W0521 22:29:18.507666 1512 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.055s user 0.017s sys 0.004s W0521 22:29:18.564595 1518 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.059s user 0.022s sys 0.000s W0521 22:29:18.588279 1507 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.177s user 0.021s sys 0.002s W0521 22:29:18.638293 1515 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.168s user 0.022s sys 0.000s W0521 22:29:18.661434 1519 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.141s user 0.020s sys 0.000s W0521 22:29:19.146560 1518 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.073s user 0.018s sys 0.000s W0521 22:29:19.182937 1512 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.081s user 0.018s sys 0.000s W0521 22:29:19.230571 1510 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.101s user 0.018s sys 0.000s I0521 22:29:19.349961 1522 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36428 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=15, attempt_no=0}) took 1399ms. Trace: I0521 22:29:19.356107 1522 rpcz_store.cc:270] 0521 22:29:17.950634 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:17.950839 (+ 205us) service_pool.cc:223] Handling call 0521 22:29:19.349897 (+1399058us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:17.953835 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:17.954714 (+ 879us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:17.954735 (+ 21us) write_op.cc:338] Acquired schema lock 0521 22:29:17.954747 (+ 12us) tablet.cc:452] Decoding operations 0521 22:29:18.070430 (+115683us) tablet.cc:474] Acquiring locks for 508 operations 0521 22:29:18.409061 (+338631us) tablet.cc:479] Locks acquired 0521 22:29:18.409081 (+ 20us) write_op.cc:186] PREPARE: Finished. 0521 22:29:18.409280 (+ 199us) write_op.cc:196] Start() 0521 22:29:18.409375 (+ 95us) write_op.cc:201] Timestamp: P: 1590100158409234 usec, L: 0 0521 22:29:18.409393 (+ 18us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:18.410813 (+ 1420us) log.cc:840] Serialized 71178 byte log entry 0521 22:29:18.538789 (+127976us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:18.539696 (+ 907us) write_op.cc:225] APPLY: Starting. 0521 22:29:19.302670 (+762974us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:19.302710 (+ 40us) write_op.cc:236] APPLY: Finished. 0521 22:29:19.311703 (+ 8993us) log.cc:840] Serialized 3067 byte log entry 0521 22:29:19.321603 (+ 9900us) write_op.cc:379] Releasing row and schema locks 0521 22:29:19.338684 (+ 17081us) write_op.cc:344] Released schema lock 0521 22:29:19.349660 (+ 10976us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":766,"num_ops":508,"prepare.queue_time_us":162,"prepare.run_cpu_time_us":72997,"prepare.run_wall_time_us":490300,"raft.queue_time_us":244,"raft.run_cpu_time_us":3319,"raft.run_wall_time_us":3939,"replication_time_us":129296,"spinlock_wait_cycles":562816}]]} W0521 22:29:19.570113 1512 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.093s user 0.019s sys 0.000s W0521 22:29:19.597034 1510 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.129s user 0.020s sys 0.000s I0521 22:29:19.759548 1531 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36428 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=17, attempt_no=0}) took 1716ms. Trace: I0521 22:29:19.768115 1531 rpcz_store.cc:270] 0521 22:29:18.042956 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:18.043088 (+ 132us) service_pool.cc:223] Handling call 0521 22:29:19.759482 (+1716394us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:18.444314 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:18.445147 (+ 833us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:18.445170 (+ 23us) write_op.cc:338] Acquired schema lock 0521 22:29:18.445183 (+ 13us) tablet.cc:452] Decoding operations 0521 22:29:18.797354 (+352171us) tablet.cc:474] Acquiring locks for 509 operations 0521 22:29:19.030921 (+233567us) tablet.cc:479] Locks acquired 0521 22:29:19.030936 (+ 15us) write_op.cc:186] PREPARE: Finished. 0521 22:29:19.031102 (+ 166us) write_op.cc:196] Start() 0521 22:29:19.031187 (+ 85us) write_op.cc:201] Timestamp: P: 1590100159031056 usec, L: 0 0521 22:29:19.031202 (+ 15us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:19.102608 (+ 71406us) log.cc:840] Serialized 71317 byte log entry 0521 22:29:19.200760 (+ 98152us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:19.201015 (+ 255us) write_op.cc:225] APPLY: Starting. 0521 22:29:19.708500 (+507485us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:19.708542 (+ 42us) write_op.cc:236] APPLY: Finished. 0521 22:29:19.711916 (+ 3374us) log.cc:840] Serialized 3073 byte log entry 0521 22:29:19.728141 (+ 16225us) write_op.cc:379] Releasing row and schema locks 0521 22:29:19.756396 (+ 28255us) write_op.cc:344] Released schema lock 0521 22:29:19.759262 (+ 2866us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":128,"num_ops":509,"prepare.queue_time_us":396484,"prepare.run_cpu_time_us":71394,"prepare.run_wall_time_us":663928,"raft.queue_time_us":118,"raft.run_cpu_time_us":1843,"raft.run_wall_time_us":10535,"replication_time_us":169463,"spinlock_wait_cycles":30080}]]} I0521 22:29:20.097263 1522 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:36428 (ReqId={client: cf7e57ef59544d6fb1124a196d50bf47, seq_no=18, attempt_no=0}) took 1463ms. Trace: I0521 22:29:20.097635 1522 rpcz_store.cc:270] 0521 22:29:18.633763 (+ 0us) service_pool.cc:164] Inserting onto call queue 0521 22:29:18.633935 (+ 172us) service_pool.cc:223] Handling call 0521 22:29:20.097200 (+1463265us) inbound_call.cc:163] Queueing success response Related trace 'op': 0521 22:29:19.108398 (+ 0us) write_op.cc:151] PREPARE: Starting. 0521 22:29:19.109068 (+ 670us) write_op.cc:335] Acquiring schema lock in shared mode 0521 22:29:19.109081 (+ 13us) write_op.cc:338] Acquired schema lock 0521 22:29:19.109088 (+ 7us) tablet.cc:452] Decoding operations 0521 22:29:19.333294 (+224206us) tablet.cc:474] Acquiring locks for 511 operations 0521 22:29:19.429842 (+ 96548us) tablet.cc:479] Locks acquired 0521 22:29:19.429861 (+ 19us) write_op.cc:186] PREPARE: Finished. 0521 22:29:19.430042 (+ 181us) write_op.cc:196] Start() 0521 22:29:19.430136 (+ 94us) write_op.cc:201] Timestamp: P: 1590100159429994 usec, L: 0 0521 22:29:19.430153 (+ 17us) op_driver.cc:321] REPLICATION: Starting. 0521 22:29:19.431614 (+ 1461us) log.cc:840] Serialized 71595 byte log entry 0521 22:29:19.573229 (+141615us) op_driver.cc:434] REPLICATION: Finished. 0521 22:29:19.573564 (+ 335us) write_op.cc:225] APPLY: Starting. 0521 22:29:20.043080 (+469516us) tablet_metrics.cc:461] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0 0521 22:29:20.043119 (+ 39us) write_op.cc:236] APPLY: Finished. 0521 22:29:20.069481 (+ 26362us) log.cc:840] Serialized 3085 byte log entry 0521 22:29:20.069597 (+ 116us) write_op.cc:379] Releasing row and schema locks 0521 22:29:20.085913 (+ 16316us) write_op.cc:344] Released schema lock 0521 22:29:20.096968 (+ 11055us) write_op.cc:260] FINISH: Updating metrics. Metrics: {"child_traces":[["op",{"apply.queue_time_us":208,"num_ops":511,"prepare.queue_time_us":457687,"prepare.run_cpu_time_us":66440,"prepare.run_wall_time_us":340827,"raft.queue_time_us":4413,"raft.run_cpu_time_us":3511,"raft.run_wall_time_us":14843,"replication_time_us":143000,"spinlock_wait_cycles":292224}]]} W0521 22:29:20.147557 1507 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P 343052a5d259480dbf4599078770bfe1: Append to log took a long time: real 0.109s user 0.017s sys 0.000s W0521 22:29:20.164208 1512 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.094s user 0.020s sys 0.000s W0521 22:29:20.167805 1515 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.069s user 0.019s sys 0.001s W0521 22:29:20.185343 1519 log.cc:914] Time spent T e7001908da8140df8082f3243384a351 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.067s user 0.020s sys 0.000s W0521 22:29:20.242264 1510 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.160s user 0.021s sys 0.000s W0521 22:29:20.583056 1510 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P e4889a5698f944459719d537988dbaea: Append to log took a long time: real 0.050s user 0.018s sys 0.000s W0521 22:29:20.602263 1512 log.cc:914] Time spent T db62be61ab374802a1d543ea858f9113 P aa747fc1bcba4e4fb55a9bce3a0c7c37: Append to log took a long time: real 0.065s user 0.020s sys 0.000s INSERT report rows total: 12288 time total: 6917.13 ms time per row: 0.562917 ms Scanner report expected rows: 12288 actual rows : 12288 W0521 22:29:21.174237 1325 debug-util.cc:402] Leaking SignalData structure 0x7b080002a800 after lost signal to thread 1312 W0521 22:29:21.175644 1325 debug-util.cc:402] Leaking SignalData structure 0x7b08000bbbc0 after lost signal to thread 1431 I0521 22:29:21.996349 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver with pid 1049 I0521 22:29:22.056763 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver with pid 1185 I0521 22:29:22.112429 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver with pid 1311 I0521 22:29:22.144636 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master with pid 986 2020-05-21T22:29:22Z chronyd exiting [ OK ] ToolTest.TestLoadgenManualFlush (15847 ms) [ RUN ] ToolTest.TestLoadgenHmsEnabled 2020-05-21T22:29:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-05-21T22:29:22Z Disabled control of system clock Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/derby.log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/tmp/dist-test-taskopyat3/thirdparty/src/hive-3.1.1/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/dist-test-taskopyat3/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Metastore connection URL: jdbc:derby:/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/metadb;create=true Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver Metastore connection User: APP Starting metastore schema initialization to 3.1.0 Initialization script hive-schema-3.1.0.derby.sql Initialization script completed schemaTool completed Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/derby.log 2020-05-21 22:29:28: Starting Hive Metastore Server Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/derby.log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/tmp/dist-test-taskopyat3/thirdparty/src/hive-3.1.1/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/dist-test-taskopyat3/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Starting hive metastore on port 0 I0521 22:29:38.262449 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --ipki_ca_key_size=768 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:42355 --webserver_interface=127.0.89.254 --webserver_port=0 --builtin_ntp_servers=127.0.89.212:43381 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:29:38.408885 1831 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:29:38.409461 1831 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:29:38.409960 1831 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:29:38.422197 1831 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=768 W0521 22:29:38.422410 1831 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:29:38.422652 1831 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0521 22:29:38.422879 1831 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0521 22:29:38.442242 1831 master_runner.cc:93] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:43381 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/wal --ipki_ca_key_size=768 --ipki_server_key_size=768 --openssl_security_level_override=0 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:42355 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.89.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:29:38.443125 1831 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:29:38.444656 1831 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:29:38.459511 1831 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:29:38.463222 1831 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:29:38.464577 1831 hybrid_clock.cc:590] HybridClock initialized: now 1590100178464522 us; error 62 us; skew 500 ppm I0521 22:29:38.466629 1831 fs_manager.cc:259] Metadata directory not provided I0521 22:29:38.466771 1831 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:29:38.467077 1831 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:29:38.470204 1831 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/data/instance: uuid: "094a180260dd4f009e2ab26992767d87" format_stamp: "Formatted at 2020-05-21 22:29:38 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:38.470997 1831 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/wal/instance: uuid: "094a180260dd4f009e2ab26992767d87" format_stamp: "Formatted at 2020-05-21 22:29:38 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:38.490568 1831 fs_manager.cc:518] Time spent creating directory manager: real 0.019s user 0.021s sys 0.000s I0521 22:29:38.498746 1831 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:29:38.498884 1831 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/wal uuid: "094a180260dd4f009e2ab26992767d87" format_stamp: "Formatted at 2020-05-21 22:29:38 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:38.499148 1831 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:29:39.275455 1831 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.750s user 0.359s sys 0.362s W0521 22:29:39.275758 1831 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.750s user 0.359s sys 0.362s I0521 22:29:39.296707 1831 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:29:39.361045 1831 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.254:42355 I0521 22:29:39.364552 1831 webserver.cc:381] Webserver started at http://127.0.89.254:38129/ using document root and password file I0521 22:29:39.365505 1831 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/master-0/data/info.pb I0521 22:29:39.367147 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master as pid 1831 I0521 22:29:39.368729 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --rpc_bind_addresses=127.0.89.193:0 --local_ip_for_outbound_sockets=127.0.89.193 --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:42355 --builtin_ntp_servers=127.0.89.212:43381 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} I0521 22:29:39.371217 1893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:39.398772 1893 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: Bootstrap starting. I0521 22:29:39.405126 1893 tablet_bootstrap.cc:596] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: No blocks or log segments found. Creating new log. I0521 22:29:39.407526 1893 log.cc:819] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: Log is configured to *not* fsync() on all Append() calls I0521 22:29:39.412887 1893 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: No bootstrap required, opened a new log I0521 22:29:39.436084 1893 raft_consensus.cc:342] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094a180260dd4f009e2ab26992767d87" member_type: VOTER } I0521 22:29:39.436496 1893 raft_consensus.cc:368] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:39.436703 1893 raft_consensus.cc:722] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 094a180260dd4f009e2ab26992767d87, State: Initialized, Role: FOLLOWER I0521 22:29:39.437346 1893 consensus_queue.cc:250] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094a180260dd4f009e2ab26992767d87" member_type: VOTER } I0521 22:29:39.437736 1893 raft_consensus.cc:382] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:39.437988 1893 raft_consensus.cc:478] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:39.438241 1893 raft_consensus.cc:2975] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:39.441483 1893 raft_consensus.cc:500] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094a180260dd4f009e2ab26992767d87" member_type: VOTER } I0521 22:29:39.442045 1893 leader_election.cc:302] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 094a180260dd4f009e2ab26992767d87; no voters: I0521 22:29:39.443760 1893 leader_election.cc:288] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [CANDIDATE]: Term 1 election: Requested vote from peers WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:29:39.511361 1894 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:29:39.511785 1894 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:29:39.512373 1894 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:29:39.522889 1894 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:29:39.523833 1894 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.89.193 I0521 22:29:39.544674 1894 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:43381 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal --ipki_server_key_size=768 --openssl_security_level_override=0 --rpc_bind_addresses=127.0.89.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:42355 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.89.193 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:29:39.545644 1894 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:29:39.547097 1894 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:29:39.562980 1894 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:29:39.565809 1894 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:29:39.567122 1894 hybrid_clock.cc:590] HybridClock initialized: now 1590100179567110 us; error 30 us; skew 500 ppm I0521 22:29:39.568956 1894 fs_manager.cc:259] Metadata directory not provided I0521 22:29:39.569118 1894 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:29:39.569453 1894 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:29:39.572691 1894 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/data/instance: uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" format_stamp: "Formatted at 2020-05-21 22:29:39 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:39.573524 1894 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal/instance: uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" format_stamp: "Formatted at 2020-05-21 22:29:39 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:39.594655 1894 fs_manager.cc:518] Time spent creating directory manager: real 0.021s user 0.021s sys 0.000s I0521 22:29:39.602730 1894 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:29:39.602891 1894 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" format_stamp: "Formatted at 2020-05-21 22:29:39 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:39.603184 1894 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0521 22:29:39.444169 1896 raft_consensus.cc:2721] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:39.677983 1896 raft_consensus.cc:683] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [term 1 LEADER]: Becoming Leader. State: Replica: 094a180260dd4f009e2ab26992767d87, State: Running, Role: LEADER I0521 22:29:39.678876 1896 consensus_queue.cc:227] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094a180260dd4f009e2ab26992767d87" member_type: VOTER } I0521 22:29:39.689716 1897 sys_catalog.cc:357] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "094a180260dd4f009e2ab26992767d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094a180260dd4f009e2ab26992767d87" member_type: VOTER } } I0521 22:29:39.690425 1897 sys_catalog.cc:360] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [sys.catalog]: This master's current role is: LEADER I0521 22:29:39.690618 1893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [sys.catalog]: configured and running, proceeding with master startup. I0521 22:29:39.692451 1910 sys_catalog.cc:357] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 094a180260dd4f009e2ab26992767d87. Latest consensus state: current_term: 1 leader_uuid: "094a180260dd4f009e2ab26992767d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "094a180260dd4f009e2ab26992767d87" member_type: VOTER } } I0521 22:29:39.693531 1910 sys_catalog.cc:360] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87 [sys.catalog]: This master's current role is: LEADER W0521 22:29:39.719439 1913 catalog_manager.cc:1201] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0521 22:29:39.719815 1913 catalog_manager.cc:667] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0521 22:29:39.720332 1912 catalog_manager.cc:1131] Loading table and tablet metadata into memory... I0521 22:29:39.733983 1912 catalog_manager.cc:1142] Initializing Kudu internal certificate authority... I0521 22:29:39.830356 1912 catalog_manager.cc:1026] Generated new certificate authority record I0521 22:29:39.833470 1912 catalog_manager.cc:1151] Loading token signing keys... I0521 22:29:39.884985 1912 catalog_manager.cc:4538] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: Generated new TSK 0 I0521 22:29:39.886330 1912 catalog_manager.cc:1161] Initializing in-progress tserver states... W0521 22:29:40.509657 1894 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.877s user 0.001s sys 0.000s W0521 22:29:40.509979 1894 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.878s user 0.002s sys 0.001s I0521 22:29:40.561259 1894 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:29:40.566344 1894 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:29:40.566596 1894 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:29:40.726295 1894 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.193:32791 I0521 22:29:40.731042 1894 webserver.cc:381] Webserver started at http://127.0.89.193:33317/ using document root and password file I0521 22:29:40.732187 1894 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/data/info.pb I0521 22:29:40.738325 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver as pid 1894 I0521 22:29:40.769148 2027 heartbeater.cc:325] Connected to a master server at 127.0.89.254:42355 I0521 22:29:40.769553 2027 heartbeater.cc:416] Registering TS with master... I0521 22:29:40.770464 2027 heartbeater.cc:465] Master 127.0.89.254:42355 requested a full tablet report, sending... I0521 22:29:40.773031 1860 ts_manager.cc:191] Registered new tserver with Master: 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193:32791) I0521 22:29:40.774147 359 external_mini_cluster.cc:699] 1 TS(s) registered with all masters I0521 22:29:40.775946 1860 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.89.193:42093 I0521 22:29:41.780299 2027 heartbeater.cc:457] Master 127.0.89.254:42355 was elected leader, sending a full tablet report... I0521 22:29:42.245200 1860 catalog_manager.cc:1501] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37262: name: "default.loadgen_auto_a520b38651484728b239941c5a736af0" schema { columns { name: "key" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 8 seed: 0 } range_schema { columns { name: "key" } } } I0521 22:29:42.318509 1961 tablet_service.cc:1232] Processing CreateTablet for tablet 83aa2bf8513e451f8306740264a3803d (table=default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED I0521 22:29:42.324849 1961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83aa2bf8513e451f8306740264a3803d. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:42.319133 1960 tablet_service.cc:1232] Processing CreateTablet for tablet 848b408ab3f442b89a009e896bedfa42 (table=default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED I0521 22:29:42.358198 1960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848b408ab3f442b89a009e896bedfa42. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:42.320482 1963 tablet_service.cc:1232] Processing CreateTablet for tablet 5fd1558bc9954796ab33160f32dadb09 (table=default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0521 22:29:42.362084 1963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fd1558bc9954796ab33160f32dadb09. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:42.321460 1964 tablet_service.cc:1232] Processing CreateTablet for tablet 12f4f5220d084576815ce1c5a257f08f (table=default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0521 22:29:42.365999 1964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f4f5220d084576815ce1c5a257f08f. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:42.321990 1959 tablet_service.cc:1232] Processing CreateTablet for tablet 01ddf7bac38243399df51c3c834e5d11 (table=default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED I0521 22:29:42.369666 1959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ddf7bac38243399df51c3c834e5d11. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:42.323951 1958 tablet_service.cc:1232] Processing CreateTablet for tablet bc3c2ebc310c49ef900e7e605859dca0 (table=default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED I0521 22:29:42.377966 1958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc3c2ebc310c49ef900e7e605859dca0. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:42.324404 1962 tablet_service.cc:1232] Processing CreateTablet for tablet 4542efb30bcd49ed9129c50322c2a46f (table=default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0521 22:29:42.382131 1962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4542efb30bcd49ed9129c50322c2a46f. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:42.329239 1957 tablet_service.cc:1232] Processing CreateTablet for tablet 242fc29f3e354eb093742447e66cfdf3 (table=default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED I0521 22:29:42.444784 1957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242fc29f3e354eb093742447e66cfdf3. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:42.445716 2052 tablet_bootstrap.cc:440] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Bootstrap starting. I0521 22:29:42.450492 2052 tablet_bootstrap.cc:596] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No blocks or log segments found. Creating new log. I0521 22:29:42.452013 2052 log.cc:819] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Log is configured to *not* fsync() on all Append() calls I0521 22:29:42.455713 2052 tablet_bootstrap.cc:440] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No bootstrap required, opened a new log I0521 22:29:42.456087 2052 ts_tablet_manager.cc:1115] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s I0521 22:29:42.469521 2052 raft_consensus.cc:342] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.470031 2052 raft_consensus.cc:368] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:42.470222 2052 raft_consensus.cc:722] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Initialized, Role: FOLLOWER I0521 22:29:42.470791 2052 consensus_queue.cc:250] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.471230 2052 raft_consensus.cc:382] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:42.471452 2052 raft_consensus.cc:478] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:42.471683 2052 raft_consensus.cc:2975] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:42.475211 2052 raft_consensus.cc:500] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.475744 2052 leader_election.cc:302] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7b9aa33cb9488eb6b9dd818c9cd33a; no voters: I0521 22:29:42.477167 2052 leader_election.cc:288] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:42.477537 2054 raft_consensus.cc:2721] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:42.479745 2052 ts_tablet_manager.cc:1144] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent starting tablet: real 0.023s user 0.019s sys 0.004s I0521 22:29:42.480238 2054 raft_consensus.cc:683] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Becoming Leader. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Running, Role: LEADER I0521 22:29:42.480655 2052 tablet_bootstrap.cc:440] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Bootstrap starting. I0521 22:29:42.480834 2054 consensus_queue.cc:227] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.484869 2052 tablet_bootstrap.cc:596] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No blocks or log segments found. Creating new log. W0521 22:29:42.486600 2028 tablet.cc:1820] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Can't schedule compaction. Clean time has not been advanced past its initial value. I0521 22:29:42.489410 1860 catalog_manager.cc:4202] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a reported cstate change: term changed from 0 to 1, leader changed from to 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193). New cstate: current_term: 1 leader_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } health_report { overall_health: HEALTHY } } } I0521 22:29:42.495354 2052 tablet_bootstrap.cc:440] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No bootstrap required, opened a new log I0521 22:29:42.495653 2052 ts_tablet_manager.cc:1115] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s I0521 22:29:42.497870 2052 raft_consensus.cc:342] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.498222 2052 raft_consensus.cc:368] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:42.498430 2052 raft_consensus.cc:722] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Initialized, Role: FOLLOWER I0521 22:29:42.498914 2052 consensus_queue.cc:250] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.499286 2052 raft_consensus.cc:382] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:42.499461 2052 raft_consensus.cc:478] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:42.499677 2052 raft_consensus.cc:2975] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:42.503605 2052 raft_consensus.cc:500] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.504087 2052 leader_election.cc:302] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7b9aa33cb9488eb6b9dd818c9cd33a; no voters: I0521 22:29:42.504449 2052 leader_election.cc:288] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:42.504576 2054 raft_consensus.cc:2721] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:42.505013 2054 raft_consensus.cc:683] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Becoming Leader. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Running, Role: LEADER I0521 22:29:42.505519 2052 ts_tablet_manager.cc:1144] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent starting tablet: real 0.010s user 0.003s sys 0.007s I0521 22:29:42.505558 2054 consensus_queue.cc:227] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.506083 2052 tablet_bootstrap.cc:440] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Bootstrap starting. I0521 22:29:42.511080 1859 catalog_manager.cc:4202] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a reported cstate change: term changed from 0 to 1, leader changed from to 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193). New cstate: current_term: 1 leader_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } health_report { overall_health: HEALTHY } } } I0521 22:29:42.514760 2052 tablet_bootstrap.cc:596] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No blocks or log segments found. Creating new log. I0521 22:29:42.519934 2052 tablet_bootstrap.cc:440] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No bootstrap required, opened a new log I0521 22:29:42.520296 2052 ts_tablet_manager.cc:1115] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s I0521 22:29:42.522147 2052 raft_consensus.cc:342] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.522547 2052 raft_consensus.cc:368] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:42.522783 2052 raft_consensus.cc:722] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Initialized, Role: FOLLOWER I0521 22:29:42.523228 2052 consensus_queue.cc:250] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.523641 2052 raft_consensus.cc:382] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:42.523833 2052 raft_consensus.cc:478] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:42.524106 2052 raft_consensus.cc:2975] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:42.527307 2052 raft_consensus.cc:500] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.527763 2052 leader_election.cc:302] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7b9aa33cb9488eb6b9dd818c9cd33a; no voters: I0521 22:29:42.528208 2052 leader_election.cc:288] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:42.528322 2054 raft_consensus.cc:2721] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:42.528666 2054 raft_consensus.cc:683] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Becoming Leader. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Running, Role: LEADER I0521 22:29:42.529189 2054 consensus_queue.cc:227] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.529438 2052 ts_tablet_manager.cc:1144] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s I0521 22:29:42.529978 2052 tablet_bootstrap.cc:440] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Bootstrap starting. I0521 22:29:42.533964 1859 catalog_manager.cc:4202] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a reported cstate change: term changed from 0 to 1, leader changed from to 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193). New cstate: current_term: 1 leader_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } health_report { overall_health: HEALTHY } } } I0521 22:29:42.534996 2052 tablet_bootstrap.cc:596] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No blocks or log segments found. Creating new log. I0521 22:29:42.540307 2052 tablet_bootstrap.cc:440] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No bootstrap required, opened a new log I0521 22:29:42.540685 2052 ts_tablet_manager.cc:1115] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s I0521 22:29:42.542551 2052 raft_consensus.cc:342] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.542991 2052 raft_consensus.cc:368] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:42.543232 2052 raft_consensus.cc:722] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Initialized, Role: FOLLOWER I0521 22:29:42.543857 2052 consensus_queue.cc:250] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.544484 2052 raft_consensus.cc:382] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:42.544754 2052 raft_consensus.cc:478] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:42.545076 2052 raft_consensus.cc:2975] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:42.548509 2052 raft_consensus.cc:500] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.548966 2052 leader_election.cc:302] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7b9aa33cb9488eb6b9dd818c9cd33a; no voters: I0521 22:29:42.549544 2052 leader_election.cc:288] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:42.549638 2054 raft_consensus.cc:2721] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:42.550935 2052 ts_tablet_manager.cc:1144] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s I0521 22:29:42.551491 2052 tablet_bootstrap.cc:440] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Bootstrap starting. I0521 22:29:42.552088 2054 raft_consensus.cc:683] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Becoming Leader. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Running, Role: LEADER I0521 22:29:42.552551 2054 consensus_queue.cc:227] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.558856 2052 tablet_bootstrap.cc:596] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No blocks or log segments found. Creating new log. I0521 22:29:42.564755 1859 catalog_manager.cc:4202] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a reported cstate change: term changed from 0 to 1, leader changed from to 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193). New cstate: current_term: 1 leader_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } health_report { overall_health: HEALTHY } } } I0521 22:29:42.565444 2052 tablet_bootstrap.cc:440] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No bootstrap required, opened a new log I0521 22:29:42.565810 2052 ts_tablet_manager.cc:1115] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.006s I0521 22:29:42.568140 2052 raft_consensus.cc:342] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.568502 2052 raft_consensus.cc:368] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:42.568816 2052 raft_consensus.cc:722] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Initialized, Role: FOLLOWER I0521 22:29:42.569578 2052 consensus_queue.cc:250] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.570334 2052 raft_consensus.cc:382] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:42.570590 2052 raft_consensus.cc:478] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:42.570891 2052 raft_consensus.cc:2975] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:42.575455 2052 raft_consensus.cc:500] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.575984 2052 leader_election.cc:302] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7b9aa33cb9488eb6b9dd818c9cd33a; no voters: I0521 22:29:42.576747 2052 leader_election.cc:288] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:42.576804 2056 raft_consensus.cc:2721] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:42.577246 2056 raft_consensus.cc:683] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Becoming Leader. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Running, Role: LEADER I0521 22:29:42.577751 2056 consensus_queue.cc:227] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.580230 2052 ts_tablet_manager.cc:1144] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s I0521 22:29:42.581033 2052 tablet_bootstrap.cc:440] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Bootstrap starting. I0521 22:29:42.588002 2052 tablet_bootstrap.cc:596] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No blocks or log segments found. Creating new log. I0521 22:29:42.587466 1859 catalog_manager.cc:4202] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a reported cstate change: term changed from 0 to 1, leader changed from to 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193). New cstate: current_term: 1 leader_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } health_report { overall_health: HEALTHY } } } I0521 22:29:42.593554 2052 tablet_bootstrap.cc:440] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No bootstrap required, opened a new log I0521 22:29:42.593940 2052 ts_tablet_manager.cc:1115] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.001s I0521 22:29:42.596449 2052 raft_consensus.cc:342] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.596944 2052 raft_consensus.cc:368] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:42.597172 2052 raft_consensus.cc:722] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Initialized, Role: FOLLOWER I0521 22:29:42.597697 2052 consensus_queue.cc:250] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.598260 2052 raft_consensus.cc:382] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:42.598536 2052 raft_consensus.cc:478] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:42.598824 2052 raft_consensus.cc:2975] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:42.603058 2052 raft_consensus.cc:500] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.603469 2052 leader_election.cc:302] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7b9aa33cb9488eb6b9dd818c9cd33a; no voters: I0521 22:29:42.603811 2052 leader_election.cc:288] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:42.603974 2056 raft_consensus.cc:2721] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:42.604650 2056 raft_consensus.cc:683] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Becoming Leader. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Running, Role: LEADER I0521 22:29:42.605023 2052 ts_tablet_manager.cc:1144] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s I0521 22:29:42.605506 2052 tablet_bootstrap.cc:440] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Bootstrap starting. I0521 22:29:42.605264 2056 consensus_queue.cc:227] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.610105 2052 tablet_bootstrap.cc:596] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No blocks or log segments found. Creating new log. I0521 22:29:42.612128 1859 catalog_manager.cc:4202] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a reported cstate change: term changed from 0 to 1, leader changed from to 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193). New cstate: current_term: 1 leader_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } health_report { overall_health: HEALTHY } } } I0521 22:29:42.616688 2052 tablet_bootstrap.cc:440] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No bootstrap required, opened a new log I0521 22:29:42.616976 2052 ts_tablet_manager.cc:1115] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s I0521 22:29:42.618695 2052 raft_consensus.cc:342] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.619177 2052 raft_consensus.cc:368] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:42.619351 2052 raft_consensus.cc:722] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Initialized, Role: FOLLOWER I0521 22:29:42.619817 2052 consensus_queue.cc:250] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.620246 2052 raft_consensus.cc:382] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:42.620491 2052 raft_consensus.cc:478] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:42.620764 2052 raft_consensus.cc:2975] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:42.624641 2052 raft_consensus.cc:500] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.625085 2052 leader_election.cc:302] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7b9aa33cb9488eb6b9dd818c9cd33a; no voters: I0521 22:29:42.625447 2052 leader_election.cc:288] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:42.625591 2056 raft_consensus.cc:2721] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:42.625950 2056 raft_consensus.cc:683] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Becoming Leader. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Running, Role: LEADER I0521 22:29:42.626498 2056 consensus_queue.cc:227] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.627033 2052 ts_tablet_manager.cc:1144] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s I0521 22:29:42.627643 2052 tablet_bootstrap.cc:440] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Bootstrap starting. I0521 22:29:42.630890 2052 tablet_bootstrap.cc:596] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No blocks or log segments found. Creating new log. I0521 22:29:42.636127 2052 tablet_bootstrap.cc:440] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: No bootstrap required, opened a new log I0521 22:29:42.636528 2052 ts_tablet_manager.cc:1115] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s I0521 22:29:42.636642 1859 catalog_manager.cc:4202] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a reported cstate change: term changed from 0 to 1, leader changed from to 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193). New cstate: current_term: 1 leader_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } health_report { overall_health: HEALTHY } } } I0521 22:29:42.638731 2052 raft_consensus.cc:342] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.639328 2052 raft_consensus.cc:368] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:42.639546 2052 raft_consensus.cc:722] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Initialized, Role: FOLLOWER I0521 22:29:42.640172 2052 consensus_queue.cc:250] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.640724 2052 raft_consensus.cc:382] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:42.640969 2052 raft_consensus.cc:478] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:42.641268 2052 raft_consensus.cc:2975] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:42.646073 2052 raft_consensus.cc:500] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.646672 2052 leader_election.cc:302] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7b9aa33cb9488eb6b9dd818c9cd33a; no voters: I0521 22:29:42.647157 2052 leader_election.cc:288] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:42.647248 2056 raft_consensus.cc:2721] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:42.647903 2056 raft_consensus.cc:683] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Becoming Leader. State: Replica: 4b7b9aa33cb9488eb6b9dd818c9cd33a, State: Running, Role: LEADER I0521 22:29:42.648461 2052 ts_tablet_manager.cc:1144] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s I0521 22:29:42.648468 2056 consensus_queue.cc:227] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } } I0521 22:29:42.653867 1859 catalog_manager.cc:4202] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a reported cstate change: term changed from 0 to 1, leader changed from to 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193). New cstate: current_term: 1 leader_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b9aa33cb9488eb6b9dd818c9cd33a" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 32791 } health_report { overall_health: HEALTHY } } } I0521 22:29:43.273299 1855 catalog_manager.cc:1999] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37262: table { table_name: "default.loadgen_auto_a520b38651484728b239941c5a736af0" } modify_external_catalogs: true I0521 22:29:43.286136 1855 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: Sending DeleteTablet for 1 replicas of tablet 12f4f5220d084576815ce1c5a257f08f I0521 22:29:43.286729 1855 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: Sending DeleteTablet for 1 replicas of tablet 5fd1558bc9954796ab33160f32dadb09 I0521 22:29:43.287240 1855 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: Sending DeleteTablet for 1 replicas of tablet 4542efb30bcd49ed9129c50322c2a46f I0521 22:29:43.287766 1957 tablet_service.cc:1276] Processing DeleteTablet for tablet 12f4f5220d084576815ce1c5a257f08f with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:43 UTC) from {username='slave'} at 127.0.0.1:37860 I0521 22:29:43.288168 1961 tablet_service.cc:1276] Processing DeleteTablet for tablet 5fd1558bc9954796ab33160f32dadb09 with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:43 UTC) from {username='slave'} at 127.0.0.1:37860 I0521 22:29:43.288301 1855 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: Sending DeleteTablet for 1 replicas of tablet 83aa2bf8513e451f8306740264a3803d I0521 22:29:43.288756 1963 tablet_service.cc:1276] Processing DeleteTablet for tablet 4542efb30bcd49ed9129c50322c2a46f with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:43 UTC) from {username='slave'} at 127.0.0.1:37860 I0521 22:29:43.288897 1855 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: Sending DeleteTablet for 1 replicas of tablet 848b408ab3f442b89a009e896bedfa42 I0521 22:29:43.289826 1855 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: Sending DeleteTablet for 1 replicas of tablet 01ddf7bac38243399df51c3c834e5d11 I0521 22:29:43.289988 1957 tablet_service.cc:1276] Processing DeleteTablet for tablet 83aa2bf8513e451f8306740264a3803d with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:43 UTC) from {username='slave'} at 127.0.0.1:37860 I0521 22:29:43.290228 2092 tablet_replica.cc:277] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: stopping tablet replica I0521 22:29:43.290376 1963 tablet_service.cc:1276] Processing DeleteTablet for tablet 848b408ab3f442b89a009e896bedfa42 with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:43 UTC) from {username='slave'} at 127.0.0.1:37860 I0521 22:29:43.290824 1855 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: Sending DeleteTablet for 1 replicas of tablet bc3c2ebc310c49ef900e7e605859dca0 I0521 22:29:43.291110 2092 raft_consensus.cc:2186] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Raft consensus shutting down. I0521 22:29:43.291149 1963 tablet_service.cc:1276] Processing DeleteTablet for tablet 01ddf7bac38243399df51c3c834e5d11 with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:43 UTC) from {username='slave'} at 127.0.0.1:37860 I0521 22:29:43.291790 2092 raft_consensus.cc:2213] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:29:43.291857 1855 catalog_manager.cc:4473] T 00000000000000000000000000000000 P 094a180260dd4f009e2ab26992767d87: Sending DeleteTablet for 1 replicas of tablet 242fc29f3e354eb093742447e66cfdf3 I0521 22:29:43.292165 1963 tablet_service.cc:1276] Processing DeleteTablet for tablet bc3c2ebc310c49ef900e7e605859dca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:43 UTC) from {username='slave'} at 127.0.0.1:37860 I0521 22:29:43.293295 1963 tablet_service.cc:1276] Processing DeleteTablet for tablet 242fc29f3e354eb093742447e66cfdf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:29:43 UTC) from {username='slave'} at 127.0.0.1:37860 I0521 22:29:43.295481 2092 ts_tablet_manager.cc:1476] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting tablet data with delete state TABLET_DATA_DELETED I0521 22:29:43.306275 2092 ts_tablet_manager.cc:1489] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3 I0521 22:29:43.306576 2092 log.cc:1189] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting WAL directory at /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal/wals/12f4f5220d084576815ce1c5a257f08f I0521 22:29:43.307288 2092 ts_tablet_manager.cc:1510] T 12f4f5220d084576815ce1c5a257f08f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting consensus metadata I0521 22:29:43.310225 2092 tablet_replica.cc:277] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: stopping tablet replica I0521 22:29:43.310336 1840 catalog_manager.cc:3571] TS 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193:32791): tablet 12f4f5220d084576815ce1c5a257f08f (table default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]) successfully deleted I0521 22:29:43.310993 2092 raft_consensus.cc:2186] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Raft consensus shutting down. I0521 22:29:43.311376 2092 raft_consensus.cc:2213] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:29:43.314545 2092 ts_tablet_manager.cc:1476] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting tablet data with delete state TABLET_DATA_DELETED I0521 22:29:43.324751 2092 ts_tablet_manager.cc:1489] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3 I0521 22:29:43.325019 2092 log.cc:1189] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting WAL directory at /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal/wals/5fd1558bc9954796ab33160f32dadb09 I0521 22:29:43.325654 2092 ts_tablet_manager.cc:1510] T 5fd1558bc9954796ab33160f32dadb09 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting consensus metadata I0521 22:29:43.328907 2092 tablet_replica.cc:277] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: stopping tablet replica I0521 22:29:43.329003 1840 catalog_manager.cc:3571] TS 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193:32791): tablet 5fd1558bc9954796ab33160f32dadb09 (table default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]) successfully deleted I0521 22:29:43.329564 2092 raft_consensus.cc:2186] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Raft consensus shutting down. I0521 22:29:43.329913 2092 raft_consensus.cc:2213] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:29:43.332864 2092 ts_tablet_manager.cc:1476] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting tablet data with delete state TABLET_DATA_DELETED I0521 22:29:43.343120 2092 ts_tablet_manager.cc:1489] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3 I0521 22:29:43.343413 2092 log.cc:1189] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting WAL directory at /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal/wals/4542efb30bcd49ed9129c50322c2a46f I0521 22:29:43.344097 2092 ts_tablet_manager.cc:1510] T 4542efb30bcd49ed9129c50322c2a46f P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting consensus metadata I0521 22:29:43.346544 2092 tablet_replica.cc:277] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a: stopping tablet replica I0521 22:29:43.346575 1840 catalog_manager.cc:3571] TS 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193:32791): tablet 4542efb30bcd49ed9129c50322c2a46f (table default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]) successfully deleted I0521 22:29:43.347185 2092 raft_consensus.cc:2186] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Raft consensus shutting down. I0521 22:29:43.347541 2092 raft_consensus.cc:2213] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:29:43.350500 2092 ts_tablet_manager.cc:1476] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting tablet data with delete state TABLET_DATA_DELETED I0521 22:29:43.360627 2092 ts_tablet_manager.cc:1489] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3 I0521 22:29:43.360922 2092 log.cc:1189] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting WAL directory at /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal/wals/83aa2bf8513e451f8306740264a3803d I0521 22:29:43.361570 2092 ts_tablet_manager.cc:1510] T 83aa2bf8513e451f8306740264a3803d P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting consensus metadata I0521 22:29:43.363881 2092 tablet_replica.cc:277] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: stopping tablet replica I0521 22:29:43.363937 1840 catalog_manager.cc:3571] TS 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193:32791): tablet 83aa2bf8513e451f8306740264a3803d (table default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]) successfully deleted I0521 22:29:43.364549 2092 raft_consensus.cc:2186] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Raft consensus shutting down. I0521 22:29:43.364899 2092 raft_consensus.cc:2213] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:29:43.367827 2092 ts_tablet_manager.cc:1476] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting tablet data with delete state TABLET_DATA_DELETED I0521 22:29:43.377085 2092 ts_tablet_manager.cc:1489] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3 I0521 22:29:43.377285 2092 log.cc:1189] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting WAL directory at /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal/wals/848b408ab3f442b89a009e896bedfa42 I0521 22:29:43.377779 2092 ts_tablet_manager.cc:1510] T 848b408ab3f442b89a009e896bedfa42 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting consensus metadata I0521 22:29:43.379559 2092 tablet_replica.cc:277] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: stopping tablet replica I0521 22:29:43.379622 1840 catalog_manager.cc:3571] TS 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193:32791): tablet 848b408ab3f442b89a009e896bedfa42 (table default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]) successfully deleted I0521 22:29:43.380051 2092 raft_consensus.cc:2186] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Raft consensus shutting down. I0521 22:29:43.380345 2092 raft_consensus.cc:2213] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:29:43.382916 2092 ts_tablet_manager.cc:1476] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting tablet data with delete state TABLET_DATA_DELETED I0521 22:29:43.389499 2092 ts_tablet_manager.cc:1489] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3 I0521 22:29:43.389731 2092 log.cc:1189] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting WAL directory at /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal/wals/01ddf7bac38243399df51c3c834e5d11 I0521 22:29:43.390228 2092 ts_tablet_manager.cc:1510] T 01ddf7bac38243399df51c3c834e5d11 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting consensus metadata I0521 22:29:43.391983 2092 tablet_replica.cc:277] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: stopping tablet replica I0521 22:29:43.392158 1840 catalog_manager.cc:3571] TS 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193:32791): tablet 01ddf7bac38243399df51c3c834e5d11 (table default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]) successfully deleted I0521 22:29:43.392489 2092 raft_consensus.cc:2186] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Raft consensus shutting down. I0521 22:29:43.392771 2092 raft_consensus.cc:2213] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:29:43.394943 2092 ts_tablet_manager.cc:1476] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting tablet data with delete state TABLET_DATA_DELETED I0521 22:29:43.401453 2092 ts_tablet_manager.cc:1489] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3 I0521 22:29:43.401661 2092 log.cc:1189] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting WAL directory at /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal/wals/bc3c2ebc310c49ef900e7e605859dca0 I0521 22:29:43.402168 2092 ts_tablet_manager.cc:1510] T bc3c2ebc310c49ef900e7e605859dca0 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting consensus metadata I0521 22:29:43.403844 2092 tablet_replica.cc:277] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: stopping tablet replica I0521 22:29:43.403956 1840 catalog_manager.cc:3571] TS 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193:32791): tablet bc3c2ebc310c49ef900e7e605859dca0 (table default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]) successfully deleted I0521 22:29:43.404326 2092 raft_consensus.cc:2186] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 LEADER]: Raft consensus shutting down. I0521 22:29:43.404619 2092 raft_consensus.cc:2213] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:29:43.406841 2092 ts_tablet_manager.cc:1476] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting tablet data with delete state TABLET_DATA_DELETED I0521 22:29:43.413287 2092 ts_tablet_manager.cc:1489] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3 I0521 22:29:43.413507 2092 log.cc:1189] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting WAL directory at /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLoadgenHmsEnabled.1590100104622499-359/minicluster-data/ts-0/wal/wals/242fc29f3e354eb093742447e66cfdf3 I0521 22:29:43.414026 2092 ts_tablet_manager.cc:1510] T 242fc29f3e354eb093742447e66cfdf3 P 4b7b9aa33cb9488eb6b9dd818c9cd33a: Deleting consensus metadata I0521 22:29:43.415879 1840 catalog_manager.cc:3571] TS 4b7b9aa33cb9488eb6b9dd818c9cd33a (127.0.89.193:32791): tablet 242fc29f3e354eb093742447e66cfdf3 (table default.loadgen_auto_a520b38651484728b239941c5a736af0 [id=d92e77f7e37841b99de0404d164a96f0]) successfully deleted I0521 22:29:44.319471 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver with pid 1894 I0521 22:29:44.344019 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master with pid 1831 Shutting down hive metastore. 2020-05-21T22:29:44Z chronyd exiting [ OK ] ToolTest.TestLoadgenHmsEnabled (22569 ms) [ RUN ] ToolTest.TestPerfTabletScan 2020-05-21T22:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-05-21T22:29:44Z Disabled control of system clock I0521 22:29:44.777721 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --ipki_ca_key_size=768 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:46765 --webserver_interface=127.0.89.254 --webserver_port=0 --builtin_ntp_servers=127.0.89.212:42971 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:29:44.926046 2100 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:29:44.926600 2100 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:29:44.927116 2100 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:29:44.939999 2100 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=768 W0521 22:29:44.940264 2100 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:29:44.940507 2100 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0521 22:29:44.940730 2100 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0521 22:29:44.959209 2100 master_runner.cc:93] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:42971 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/wal --ipki_ca_key_size=768 --ipki_server_key_size=768 --openssl_security_level_override=0 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:46765 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.89.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:29:44.960103 2100 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:29:44.961583 2100 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:29:44.976486 2100 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:29:44.980196 2100 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:29:44.981551 2100 hybrid_clock.cc:590] HybridClock initialized: now 1590100184981496 us; error 61 us; skew 500 ppm I0521 22:29:44.983592 2100 fs_manager.cc:259] Metadata directory not provided I0521 22:29:44.983736 2100 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:29:44.984087 2100 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:29:44.987190 2100 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/data/instance: uuid: "d3706b80d38444ca81c41b4042787745" format_stamp: "Formatted at 2020-05-21 22:29:44 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:44.987977 2100 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/wal/instance: uuid: "d3706b80d38444ca81c41b4042787745" format_stamp: "Formatted at 2020-05-21 22:29:44 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:45.007642 2100 fs_manager.cc:518] Time spent creating directory manager: real 0.019s user 0.014s sys 0.004s I0521 22:29:45.015704 2100 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:29:45.015857 2100 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/wal uuid: "d3706b80d38444ca81c41b4042787745" format_stamp: "Formatted at 2020-05-21 22:29:44 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:45.016144 2100 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:29:45.764956 2100 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.722s user 0.275s sys 0.447s W0521 22:29:45.765259 2100 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.723s user 0.275s sys 0.447s I0521 22:29:45.798774 2100 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:29:45.870369 2100 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.254:46765 I0521 22:29:45.875396 2100 webserver.cc:381] Webserver started at http://127.0.89.254:43341/ using document root and password file I0521 22:29:45.876839 2100 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/master-0/data/info.pb I0521 22:29:45.881561 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master as pid 2100 I0521 22:29:45.881919 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:45.884177 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --rpc_bind_addresses=127.0.89.193:0 --local_ip_for_outbound_sockets=127.0.89.193 --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:46765 --builtin_ntp_servers=127.0.89.212:42971 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} I0521 22:29:45.906474 2163 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745: Bootstrap starting. I0521 22:29:45.913311 2163 tablet_bootstrap.cc:596] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745: No blocks or log segments found. Creating new log. I0521 22:29:45.915716 2163 log.cc:819] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745: Log is configured to *not* fsync() on all Append() calls I0521 22:29:45.921182 2163 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745: No bootstrap required, opened a new log I0521 22:29:45.945971 2163 raft_consensus.cc:342] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3706b80d38444ca81c41b4042787745" member_type: VOTER } I0521 22:29:45.946519 2163 raft_consensus.cc:368] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:45.946763 2163 raft_consensus.cc:722] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3706b80d38444ca81c41b4042787745, State: Initialized, Role: FOLLOWER I0521 22:29:45.947530 2163 consensus_queue.cc:250] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3706b80d38444ca81c41b4042787745" member_type: VOTER } I0521 22:29:45.948107 2163 raft_consensus.cc:382] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:45.948385 2163 raft_consensus.cc:478] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:45.948702 2163 raft_consensus.cc:2975] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:45.953657 2163 raft_consensus.cc:500] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3706b80d38444ca81c41b4042787745" member_type: VOTER } I0521 22:29:45.954433 2163 leader_election.cc:302] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3706b80d38444ca81c41b4042787745; no voters: I0521 22:29:45.956620 2166 raft_consensus.cc:2721] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:45.958065 2163 leader_election.cc:288] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:46.074198 2166 raft_consensus.cc:683] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [term 1 LEADER]: Becoming Leader. State: Replica: d3706b80d38444ca81c41b4042787745, State: Running, Role: LEADER I0521 22:29:46.075093 2166 consensus_queue.cc:227] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3706b80d38444ca81c41b4042787745" member_type: VOTER } I0521 22:29:46.082160 2167 sys_catalog.cc:357] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3706b80d38444ca81c41b4042787745. Latest consensus state: current_term: 1 leader_uuid: "d3706b80d38444ca81c41b4042787745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3706b80d38444ca81c41b4042787745" member_type: VOTER } } I0521 22:29:46.082712 2167 sys_catalog.cc:360] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [sys.catalog]: This master's current role is: LEADER I0521 22:29:46.082325 2168 sys_catalog.cc:357] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3706b80d38444ca81c41b4042787745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3706b80d38444ca81c41b4042787745" member_type: VOTER } } I0521 22:29:46.083015 2168 sys_catalog.cc:360] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [sys.catalog]: This master's current role is: LEADER I0521 22:29:46.084740 2163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745 [sys.catalog]: configured and running, proceeding with master startup. WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:29:46.084787 2164 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:29:46.085501 2164 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:29:46.086380 2164 flags.cc:403] Enabled unsafe flag: --never_fsync=true I0521 22:29:46.086666 2170 catalog_manager.cc:1131] Loading table and tablet metadata into memory... I0521 22:29:46.093811 2173 catalog_manager.cc:609] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0521 22:29:46.099651 2170 catalog_manager.cc:1142] Initializing Kudu internal certificate authority... W0521 22:29:46.099700 2164 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:29:46.101253 2164 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.89.193 I0521 22:29:46.130321 2164 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:42971 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/wal --ipki_server_key_size=768 --openssl_security_level_override=0 --rpc_bind_addresses=127.0.89.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:46765 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.89.193 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:29:46.131798 2164 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:29:46.133996 2164 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:29:46.150919 2170 catalog_manager.cc:1026] Generated new certificate authority record I0521 22:29:46.152654 2170 catalog_manager.cc:1151] Loading token signing keys... I0521 22:29:46.163547 2164 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:29:46.167629 2164 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:29:46.169085 2164 hybrid_clock.cc:590] HybridClock initialized: now 1590100186169025 us; error 76 us; skew 500 ppm I0521 22:29:46.171607 2164 fs_manager.cc:259] Metadata directory not provided I0521 22:29:46.171811 2164 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:29:46.172330 2164 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:29:46.176517 2170 catalog_manager.cc:4538] T 00000000000000000000000000000000 P d3706b80d38444ca81c41b4042787745: Generated new TSK 0 I0521 22:29:46.177297 2170 catalog_manager.cc:1161] Initializing in-progress tserver states... I0521 22:29:46.177139 2164 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/data/instance: uuid: "a9dab01160ba42c3b3131c7222096a7b" format_stamp: "Formatted at 2020-05-21 22:29:46 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:46.178326 2164 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/wal/instance: uuid: "a9dab01160ba42c3b3131c7222096a7b" format_stamp: "Formatted at 2020-05-21 22:29:46 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:46.217653 2164 fs_manager.cc:518] Time spent creating directory manager: real 0.039s user 0.040s sys 0.000s I0521 22:29:46.232321 2164 fs_manager.cc:433] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0521 22:29:46.232559 2164 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/wal uuid: "a9dab01160ba42c3b3131c7222096a7b" format_stamp: "Formatted at 2020-05-21 22:29:46 on dist-test-slave-dist-test-slave-jp1g" I0521 22:29:46.232969 2164 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:29:47.186486 2164 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.906s user 0.381s sys 0.520s W0521 22:29:47.186712 2164 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.906s user 0.381s sys 0.520s I0521 22:29:47.201740 2164 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:29:47.210057 2164 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:29:47.210214 2164 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:29:47.608525 2164 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.193:44821 I0521 22:29:47.616590 2164 webserver.cc:381] Webserver started at http://127.0.89.193:33765/ using document root and password file I0521 22:29:47.617759 2164 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestPerfTabletScan.1590100104622499-359/minicluster-data/ts-0/data/info.pb I0521 22:29:47.622370 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver as pid 2164 I0521 22:29:47.653323 2296 heartbeater.cc:325] Connected to a master server at 127.0.89.254:46765 I0521 22:29:47.653702 2296 heartbeater.cc:416] Registering TS with master... I0521 22:29:47.654428 2296 heartbeater.cc:465] Master 127.0.89.254:46765 requested a full tablet report, sending... I0521 22:29:47.657124 2130 ts_manager.cc:191] Registered new tserver with Master: a9dab01160ba42c3b3131c7222096a7b (127.0.89.193:44821) I0521 22:29:47.659360 359 external_mini_cluster.cc:699] 1 TS(s) registered with all masters I0521 22:29:47.660064 2130 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.89.193:55903 I0521 22:29:47.705260 2130 catalog_manager.cc:1501] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40210: name: "perf.tablet_scan" schema { columns { name: "key" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "bool_val" type: BOOL is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int8_val" type: INT8 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int16_val" type: INT16 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int32_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int64_val" type: INT64 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "float_val" type: FLOAT is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "double_val" type: DOUBLE is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "decimal32_val" type: DECIMAL32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 type_attributes { precision: 9 scale: 9 } } columns { name: "decimal64_val" type: DECIMAL64 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 type_attributes { precision: 18 scale: 2 } } columns { name: "decimal128_val" type: DECIMAL128 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 type_attributes { precision: 38 scale: 0 } } columns { name: "unixtime_micros_val" type: UNIXTIME_MICROS is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "binary_val" type: BINARY is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 2 seed: 0 } } I0521 22:29:47.783131 2233 tablet_service.cc:1232] Processing CreateTablet for tablet 1cb4bafaa19d4ef78572eb91ea9759f8 (table=perf.tablet_scan [id=987aff7ff5ee4885bef9383653de689a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0521 22:29:47.784605 2233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cb4bafaa19d4ef78572eb91ea9759f8. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:47.783457 2232 tablet_service.cc:1232] Processing CreateTablet for tablet 2e505c82cd05412c8169b661e4933c6d (table=perf.tablet_scan [id=987aff7ff5ee4885bef9383653de689a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0521 22:29:47.829340 2232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e505c82cd05412c8169b661e4933c6d. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:29:47.831650 2313 tablet_bootstrap.cc:440] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b: Bootstrap starting. I0521 22:29:47.838368 2313 tablet_bootstrap.cc:596] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b: No blocks or log segments found. Creating new log. I0521 22:29:47.840920 2313 log.cc:819] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b: Log is configured to *not* fsync() on all Append() calls I0521 22:29:47.845337 2313 tablet_bootstrap.cc:440] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b: No bootstrap required, opened a new log I0521 22:29:47.845683 2313 ts_tablet_manager.cc:1115] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s I0521 22:29:47.858891 2313 raft_consensus.cc:342] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dab01160ba42c3b3131c7222096a7b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 44821 } } I0521 22:29:47.859606 2313 raft_consensus.cc:368] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:47.859863 2313 raft_consensus.cc:722] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dab01160ba42c3b3131c7222096a7b, State: Initialized, Role: FOLLOWER I0521 22:29:47.860515 2313 consensus_queue.cc:250] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dab01160ba42c3b3131c7222096a7b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 44821 } } I0521 22:29:47.860956 2313 raft_consensus.cc:382] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:47.861186 2313 raft_consensus.cc:478] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:47.861444 2313 raft_consensus.cc:2975] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:47.864981 2313 raft_consensus.cc:500] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dab01160ba42c3b3131c7222096a7b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 44821 } } I0521 22:29:47.865567 2313 leader_election.cc:302] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9dab01160ba42c3b3131c7222096a7b; no voters: I0521 22:29:47.867224 2315 raft_consensus.cc:2721] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:47.868142 2313 leader_election.cc:288] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:47.870077 2315 raft_consensus.cc:683] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [term 1 LEADER]: Becoming Leader. State: Replica: a9dab01160ba42c3b3131c7222096a7b, State: Running, Role: LEADER I0521 22:29:47.870851 2296 heartbeater.cc:457] Master 127.0.89.254:46765 was elected leader, sending a full tablet report... I0521 22:29:47.871038 2315 consensus_queue.cc:227] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dab01160ba42c3b3131c7222096a7b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 44821 } } I0521 22:29:47.877998 2313 ts_tablet_manager.cc:1144] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b: Time spent starting tablet: real 0.032s user 0.020s sys 0.011s I0521 22:29:47.878738 2313 tablet_bootstrap.cc:440] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b: Bootstrap starting. I0521 22:29:47.886665 2313 tablet_bootstrap.cc:596] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b: No blocks or log segments found. Creating new log. I0521 22:29:47.886720 2130 catalog_manager.cc:4202] T 1cb4bafaa19d4ef78572eb91ea9759f8 P a9dab01160ba42c3b3131c7222096a7b reported cstate change: term changed from 0 to 1, leader changed from to a9dab01160ba42c3b3131c7222096a7b (127.0.89.193). New cstate: current_term: 1 leader_uuid: "a9dab01160ba42c3b3131c7222096a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dab01160ba42c3b3131c7222096a7b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 44821 } health_report { overall_health: HEALTHY } } } I0521 22:29:47.894315 2313 tablet_bootstrap.cc:440] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b: No bootstrap required, opened a new log I0521 22:29:47.894680 2313 ts_tablet_manager.cc:1115] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.005s I0521 22:29:47.896546 2313 raft_consensus.cc:342] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dab01160ba42c3b3131c7222096a7b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 44821 } } I0521 22:29:47.896957 2313 raft_consensus.cc:368] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:29:47.897137 2313 raft_consensus.cc:722] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dab01160ba42c3b3131c7222096a7b, State: Initialized, Role: FOLLOWER I0521 22:29:47.897619 2313 consensus_queue.cc:250] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dab01160ba42c3b3131c7222096a7b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 44821 } } I0521 22:29:47.898012 2313 raft_consensus.cc:382] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:29:47.898248 2313 raft_consensus.cc:478] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:29:47.898576 2313 raft_consensus.cc:2975] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [term 0 FOLLOWER]: Advancing to term 1 I0521 22:29:47.901908 2313 raft_consensus.cc:500] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dab01160ba42c3b3131c7222096a7b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 44821 } } I0521 22:29:47.902542 2313 leader_election.cc:302] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9dab01160ba42c3b3131c7222096a7b; no voters: I0521 22:29:47.903072 2313 leader_election.cc:288] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:29:47.903223 2315 raft_consensus.cc:2721] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:29:47.903975 2315 raft_consensus.cc:683] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [term 1 LEADER]: Becoming Leader. State: Replica: a9dab01160ba42c3b3131c7222096a7b, State: Running, Role: LEADER I0521 22:29:47.904686 2315 consensus_queue.cc:227] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dab01160ba42c3b3131c7222096a7b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 44821 } } I0521 22:29:47.907857 2313 ts_tablet_manager.cc:1144] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s I0521 22:29:47.911069 2130 catalog_manager.cc:4202] T 2e505c82cd05412c8169b661e4933c6d P a9dab01160ba42c3b3131c7222096a7b reported cstate change: term changed from 0 to 1, leader changed from to a9dab01160ba42c3b3131c7222096a7b (127.0.89.193). New cstate: current_term: 1 leader_uuid: "a9dab01160ba42c3b3131c7222096a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dab01160ba42c3b3131c7222096a7b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 44821 } health_report { overall_health: HEALTHY } } } W0521 22:29:48.146955 2322 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:29:48.147397 2322 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:29:49.009662 2322 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.791s user 0.001s sys 0.001s W0521 22:29:49.009865 2322 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.791s user 0.001s sys 0.001s Using table 'perf.tablet_scan' INSERT report rows total: 2000 time total: 718.279 ms time per row: 0.359139 ms W0521 22:29:49.832731 2328 outbound_call.cc:285] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.89.193:44821, user_credentials={real_user=slave}} blocked reactor thread for 58971.8us I0521 22:29:50.807451 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver with pid 2164 I0521 22:29:50.836561 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master with pid 2100 2020-05-21T22:30:03Z chronyd exiting [ OK ] ToolTest.TestPerfTabletScan (18652 ms) [ RUN ] ToolTest.TestLocalReplicaDeleteMultiple I0521 22:30:03.409620 359 internal_mini_cluster.cc:145] Creating distributed mini masters. Addrs: 127.0.89.254:44861 I0521 22:30:03.410622 359 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:30:03.411679 359 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:30:03.428217 359 hybrid_clock.cc:243] auto-selected time source: system_unsync W0521 22:30:03.428437 359 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters. I0521 22:30:03.428601 359 hybrid_clock.cc:590] HybridClock initialized: now 1590100203428575 us; error 0 us; skew 500 ppm I0521 22:30:03.429735 359 fs_manager.cc:259] Metadata directory not provided I0521 22:30:03.429896 359 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:30:03.430168 359 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:30:03.431391 359 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/master-0-root/instance: uuid: "47ff8aad212b4c20850c8579a322c143" format_stamp: "Formatted at 2020-05-21 22:30:03 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:03.451213 359 fs_manager.cc:518] Time spent creating directory manager: real 0.019s user 0.017s sys 0.004s I0521 22:30:03.458099 359 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:30:03.458272 359 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/master-0-root uuid: "47ff8aad212b4c20850c8579a322c143" format_stamp: "Formatted at 2020-05-21 22:30:03 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:03.458546 359 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/master-0-root metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0521 22:30:03.475412 359 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:30:03.527745 359 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.254:44861 I0521 22:30:03.530854 359 webserver.cc:381] Webserver started at http://127.0.89.254:38667/ using document root and password file I0521 22:30:03.533741 2441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:03.543177 2441 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143: Bootstrap starting. I0521 22:30:03.547410 2441 tablet_bootstrap.cc:596] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143: No blocks or log segments found. Creating new log. I0521 22:30:03.548965 2441 log.cc:819] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143: Log is configured to *not* fsync() on all Append() calls I0521 22:30:03.552780 2441 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143: No bootstrap required, opened a new log I0521 22:30:03.566068 2441 raft_consensus.cc:342] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ff8aad212b4c20850c8579a322c143" member_type: VOTER } I0521 22:30:03.566495 2441 raft_consensus.cc:368] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:03.566673 2441 raft_consensus.cc:722] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ff8aad212b4c20850c8579a322c143, State: Initialized, Role: FOLLOWER I0521 22:30:03.567260 2441 consensus_queue.cc:250] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ff8aad212b4c20850c8579a322c143" member_type: VOTER } I0521 22:30:03.567679 2441 raft_consensus.cc:382] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:03.567885 2441 raft_consensus.cc:478] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:03.568138 2441 raft_consensus.cc:2975] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:03.571199 2441 raft_consensus.cc:500] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ff8aad212b4c20850c8579a322c143" member_type: VOTER } I0521 22:30:03.571782 2441 leader_election.cc:302] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47ff8aad212b4c20850c8579a322c143; no voters: I0521 22:30:03.573455 2441 leader_election.cc:288] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:03.573822 2443 raft_consensus.cc:2721] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:04.954149 2443 raft_consensus.cc:683] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [term 1 LEADER]: Becoming Leader. State: Replica: 47ff8aad212b4c20850c8579a322c143, State: Running, Role: LEADER W0521 22:30:04.954586 2441 thread.cc:616] raft [worker] (thread pool) Time spent creating pthread: real 1.380s user 0.529s sys 0.797s W0521 22:30:04.954967 2441 thread.cc:582] raft [worker] (thread pool) Time spent starting thread: real 1.381s user 0.529s sys 0.797s I0521 22:30:04.955472 2443 consensus_queue.cc:227] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ff8aad212b4c20850c8579a322c143" member_type: VOTER } I0521 22:30:04.966365 2441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [sys.catalog]: configured and running, proceeding with master startup. I0521 22:30:04.965961 2447 sys_catalog.cc:357] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47ff8aad212b4c20850c8579a322c143. Latest consensus state: current_term: 1 leader_uuid: "47ff8aad212b4c20850c8579a322c143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ff8aad212b4c20850c8579a322c143" member_type: VOTER } } I0521 22:30:04.966800 2447 sys_catalog.cc:360] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [sys.catalog]: This master's current role is: LEADER I0521 22:30:04.966531 2444 sys_catalog.cc:357] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47ff8aad212b4c20850c8579a322c143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47ff8aad212b4c20850c8579a322c143" member_type: VOTER } } I0521 22:30:04.967407 2444 sys_catalog.cc:360] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [sys.catalog]: This master's current role is: LEADER I0521 22:30:04.970005 359 internal_mini_cluster.cc:165] Waiting to initialize catalog manager on master 0 W0521 22:30:04.980154 2449 catalog_manager.cc:1201] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0521 22:30:04.980554 2449 catalog_manager.cc:667] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0521 22:30:04.981029 2450 catalog_manager.cc:1131] Loading table and tablet metadata into memory... I0521 22:30:04.987768 2450 catalog_manager.cc:1142] Initializing Kudu internal certificate authority... I0521 22:30:05.045461 2450 catalog_manager.cc:1026] Generated new certificate authority record I0521 22:30:05.048125 2450 catalog_manager.cc:1151] Loading token signing keys... I0521 22:30:05.071601 2450 catalog_manager.cc:4538] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143: Generated new TSK 0 I0521 22:30:05.072623 2450 catalog_manager.cc:1161] Initializing in-progress tserver states... I0521 22:30:05.102913 359 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:30:05.118731 359 hybrid_clock.cc:243] auto-selected time source: system_unsync W0521 22:30:05.118894 359 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters. I0521 22:30:05.119042 359 hybrid_clock.cc:590] HybridClock initialized: now 1590100205119026 us; error 0 us; skew 500 ppm I0521 22:30:05.121176 359 fs_manager.cc:259] Metadata directory not provided I0521 22:30:05.121337 359 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:30:05.121616 359 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:30:05.122803 359 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/ts-0-root/instance: uuid: "0910ecf09a21464eb0366a9258316f92" format_stamp: "Formatted at 2020-05-21 22:30:05 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:05.145967 359 fs_manager.cc:518] Time spent creating directory manager: real 0.023s user 0.017s sys 0.007s I0521 22:30:05.153926 359 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0521 22:30:05.154078 359 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/ts-0-root uuid: "0910ecf09a21464eb0366a9258316f92" format_stamp: "Formatted at 2020-05-21 22:30:05 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:05.154356 359 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0521 22:30:05.205652 359 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:30:05.205843 359 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:30:05.340373 359 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.193:36477 I0521 22:30:05.343741 359 webserver.cc:381] Webserver started at http://127.0.89.193:41991/ using document root and password file I0521 22:30:05.365321 2518 heartbeater.cc:325] Connected to a master server at 127.0.89.254:44861 I0521 22:30:05.365645 2518 heartbeater.cc:416] Registering TS with master... I0521 22:30:05.366303 2518 heartbeater.cc:465] Master 127.0.89.254:44861 requested a full tablet report, sending... I0521 22:30:05.368721 2408 ts_manager.cc:191] Registered new tserver with Master: 0910ecf09a21464eb0366a9258316f92 (127.0.89.193:36477) I0521 22:30:05.369124 359 internal_mini_cluster.cc:328] 1 TS(s) registered with all masters after 0.022294707s I0521 22:30:05.371850 2408 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43614 I0521 22:30:05.373600 359 test_util.cc:232] Using random seed: 233179269 I0521 22:30:05.412448 2408 catalog_manager.cc:1501] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43616: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s" indirect_data: "" } partition_schema { range_schema { columns { name: "key" } } } I0521 22:30:05.474156 2483 tablet_service.cc:1232] Processing CreateTablet for tablet 0e54106c9f2b41ddbd5598e955f7f2b9 (table=test-workload [id=be2be18755a04d98a14932092b5d8230]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0521 22:30:05.474301 2482 tablet_service.cc:1232] Processing CreateTablet for tablet 0c6a8f75ffd44c92935ce8c24bb92a8e (table=test-workload [id=be2be18755a04d98a14932092b5d8230]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0521 22:30:05.475423 2483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e54106c9f2b41ddbd5598e955f7f2b9. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:05.477579 2480 tablet_service.cc:1232] Processing CreateTablet for tablet 4199b9e1b41947a8bde6f17bd6245deb (table=test-workload [id=be2be18755a04d98a14932092b5d8230]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0521 22:30:05.479655 2484 tablet_service.cc:1232] Processing CreateTablet for tablet d6e9964553ec42c6aa013b2fa01417a5 (table=test-workload [id=be2be18755a04d98a14932092b5d8230]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0521 22:30:05.480698 2480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4199b9e1b41947a8bde6f17bd6245deb. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:05.481163 2481 tablet_service.cc:1232] Processing CreateTablet for tablet a894f47d27674d978d208a696539c513 (table=test-workload [id=be2be18755a04d98a14932092b5d8230]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0521 22:30:05.483100 2481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a894f47d27674d978d208a696539c513. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:05.485837 2479 tablet_service.cc:1232] Processing CreateTablet for tablet 98f91e535b574584b4c783a29cdeef7d (table=test-workload [id=be2be18755a04d98a14932092b5d8230]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0521 22:30:05.487241 2479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f91e535b574584b4c783a29cdeef7d. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:05.486464 2478 tablet_service.cc:1232] Processing CreateTablet for tablet 35d79bbc87724c8a8e1d7617727056bf (table=test-workload [id=be2be18755a04d98a14932092b5d8230]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0521 22:30:05.490686 2477 tablet_service.cc:1232] Processing CreateTablet for tablet a29148539a414d429508dcb4d038f2bc (table=test-workload [id=be2be18755a04d98a14932092b5d8230]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0521 22:30:05.491636 2485 tablet_service.cc:1232] Processing CreateTablet for tablet 550f3367b26d46b892a6c07eb4981bca (table=test-workload [id=be2be18755a04d98a14932092b5d8230]), partition=RANGE (key) PARTITION VALUES < 214748364 I0521 22:30:05.492668 2478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d79bbc87724c8a8e1d7617727056bf. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:05.493237 2485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550f3367b26d46b892a6c07eb4981bca. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:05.498123 2476 tablet_service.cc:1232] Processing CreateTablet for tablet b46d06fd7afc4a64aca392c75116e7ac (table=test-workload [id=be2be18755a04d98a14932092b5d8230]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0521 22:30:05.499877 2476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b46d06fd7afc4a64aca392c75116e7ac. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:05.583988 2484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e9964553ec42c6aa013b2fa01417a5. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:05.588301 2477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a29148539a414d429508dcb4d038f2bc. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:05.589283 2482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6a8f75ffd44c92935ce8c24bb92a8e. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:05.616559 2530 tablet_bootstrap.cc:440] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92: Bootstrap starting. I0521 22:30:05.620239 2530 tablet_bootstrap.cc:596] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92: No blocks or log segments found. Creating new log. I0521 22:30:05.626850 2530 tablet_bootstrap.cc:440] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92: No bootstrap required, opened a new log I0521 22:30:05.627166 2530 ts_tablet_manager.cc:1115] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s I0521 22:30:05.629106 2530 raft_consensus.cc:342] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.629478 2530 raft_consensus.cc:368] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:05.629676 2530 raft_consensus.cc:722] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Initialized, Role: FOLLOWER I0521 22:30:05.630190 2530 consensus_queue.cc:250] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.630625 2530 raft_consensus.cc:382] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:05.630834 2530 raft_consensus.cc:478] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:05.631075 2530 raft_consensus.cc:2975] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:05.634536 2530 raft_consensus.cc:500] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.635002 2530 leader_election.cc:302] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0910ecf09a21464eb0366a9258316f92; no voters: I0521 22:30:05.636137 2532 raft_consensus.cc:2721] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:05.636839 2530 leader_election.cc:288] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:05.638892 2532 raft_consensus.cc:683] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Becoming Leader. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Running, Role: LEADER I0521 22:30:05.640213 2530 ts_tablet_manager.cc:1144] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s I0521 22:30:05.640341 2518 heartbeater.cc:457] Master 127.0.89.254:44861 was elected leader, sending a full tablet report... I0521 22:30:05.641233 2532 consensus_queue.cc:227] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.642109 2530 tablet_bootstrap.cc:440] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92: Bootstrap starting. I0521 22:30:05.648350 2530 tablet_bootstrap.cc:596] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92: No blocks or log segments found. Creating new log. I0521 22:30:05.654817 2408 catalog_manager.cc:4202] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 reported cstate change: term changed from 0 to 1, leader changed from to 0910ecf09a21464eb0366a9258316f92 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "0910ecf09a21464eb0366a9258316f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } health_report { overall_health: HEALTHY } } } I0521 22:30:05.665611 2530 tablet_bootstrap.cc:440] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92: No bootstrap required, opened a new log I0521 22:30:05.665932 2530 ts_tablet_manager.cc:1115] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.006s I0521 22:30:05.667745 2530 raft_consensus.cc:342] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.668292 2530 raft_consensus.cc:368] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:05.668530 2530 raft_consensus.cc:722] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Initialized, Role: FOLLOWER I0521 22:30:05.669080 2530 consensus_queue.cc:250] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.669509 2530 raft_consensus.cc:382] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:05.669708 2530 raft_consensus.cc:478] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:05.669940 2530 raft_consensus.cc:2975] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:05.673974 2530 raft_consensus.cc:500] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.674687 2530 leader_election.cc:302] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0910ecf09a21464eb0366a9258316f92; no voters: I0521 22:30:05.675433 2532 raft_consensus.cc:2721] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:05.675801 2532 raft_consensus.cc:683] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Becoming Leader. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Running, Role: LEADER I0521 22:30:05.676383 2532 consensus_queue.cc:227] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.681694 2530 leader_election.cc:288] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:05.683204 2530 ts_tablet_manager.cc:1144] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92: Time spent starting tablet: real 0.017s user 0.011s sys 0.000s I0521 22:30:05.683776 2530 tablet_bootstrap.cc:440] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92: Bootstrap starting. I0521 22:30:05.686517 2408 catalog_manager.cc:4202] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 reported cstate change: term changed from 0 to 1, leader changed from to 0910ecf09a21464eb0366a9258316f92 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "0910ecf09a21464eb0366a9258316f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } health_report { overall_health: HEALTHY } } } I0521 22:30:05.688967 2530 tablet_bootstrap.cc:596] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92: No blocks or log segments found. Creating new log. I0521 22:30:05.693662 2530 tablet_bootstrap.cc:440] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92: No bootstrap required, opened a new log I0521 22:30:05.694063 2530 ts_tablet_manager.cc:1115] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s I0521 22:30:05.696403 2530 raft_consensus.cc:342] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.696925 2530 raft_consensus.cc:368] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:05.697238 2530 raft_consensus.cc:722] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Initialized, Role: FOLLOWER I0521 22:30:05.697794 2530 consensus_queue.cc:250] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.698253 2530 raft_consensus.cc:382] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:05.698446 2530 raft_consensus.cc:478] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:05.698671 2530 raft_consensus.cc:2975] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:05.702733 2530 raft_consensus.cc:500] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.703294 2530 leader_election.cc:302] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0910ecf09a21464eb0366a9258316f92; no voters: I0521 22:30:05.703742 2530 leader_election.cc:288] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:05.703877 2532 raft_consensus.cc:2721] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:05.704324 2532 raft_consensus.cc:683] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Becoming Leader. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Running, Role: LEADER I0521 22:30:05.704870 2532 consensus_queue.cc:227] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.705411 2530 ts_tablet_manager.cc:1144] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s I0521 22:30:05.706117 2530 tablet_bootstrap.cc:440] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92: Bootstrap starting. I0521 22:30:05.712755 2408 catalog_manager.cc:4202] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 reported cstate change: term changed from 0 to 1, leader changed from to 0910ecf09a21464eb0366a9258316f92 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "0910ecf09a21464eb0366a9258316f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } health_report { overall_health: HEALTHY } } } I0521 22:30:05.715344 2530 tablet_bootstrap.cc:596] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92: No blocks or log segments found. Creating new log. I0521 22:30:05.723415 2530 tablet_bootstrap.cc:440] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92: No bootstrap required, opened a new log I0521 22:30:05.723709 2530 ts_tablet_manager.cc:1115] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s I0521 22:30:05.725469 2530 raft_consensus.cc:342] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.725824 2530 raft_consensus.cc:368] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:05.726003 2530 raft_consensus.cc:722] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Initialized, Role: FOLLOWER I0521 22:30:05.726418 2530 consensus_queue.cc:250] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.726819 2530 raft_consensus.cc:382] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:05.727013 2530 raft_consensus.cc:478] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:05.727229 2530 raft_consensus.cc:2975] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:05.730408 2530 raft_consensus.cc:500] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.730850 2530 leader_election.cc:302] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0910ecf09a21464eb0366a9258316f92; no voters: I0521 22:30:05.731231 2530 leader_election.cc:288] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:05.731390 2532 raft_consensus.cc:2721] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:05.732067 2532 raft_consensus.cc:683] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Becoming Leader. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Running, Role: LEADER I0521 22:30:05.732663 2530 ts_tablet_manager.cc:1144] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s I0521 22:30:05.732650 2532 consensus_queue.cc:227] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.733242 2530 tablet_bootstrap.cc:440] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92: Bootstrap starting. I0521 22:30:05.738765 2530 tablet_bootstrap.cc:596] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92: No blocks or log segments found. Creating new log. I0521 22:30:05.738911 2408 catalog_manager.cc:4202] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 reported cstate change: term changed from 0 to 1, leader changed from to 0910ecf09a21464eb0366a9258316f92 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "0910ecf09a21464eb0366a9258316f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } health_report { overall_health: HEALTHY } } } I0521 22:30:05.744458 2530 tablet_bootstrap.cc:440] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92: No bootstrap required, opened a new log I0521 22:30:05.744874 2530 ts_tablet_manager.cc:1115] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s I0521 22:30:05.747344 2530 raft_consensus.cc:342] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.747834 2530 raft_consensus.cc:368] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:05.748097 2530 raft_consensus.cc:722] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Initialized, Role: FOLLOWER I0521 22:30:05.748661 2530 consensus_queue.cc:250] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.749260 2530 raft_consensus.cc:382] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:05.749567 2530 raft_consensus.cc:478] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:05.749872 2530 raft_consensus.cc:2975] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:05.754914 2530 raft_consensus.cc:500] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.755527 2530 leader_election.cc:302] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0910ecf09a21464eb0366a9258316f92; no voters: I0521 22:30:05.756093 2530 leader_election.cc:288] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:05.756222 2532 raft_consensus.cc:2721] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:05.756721 2532 raft_consensus.cc:683] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Becoming Leader. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Running, Role: LEADER I0521 22:30:05.757359 2532 consensus_queue.cc:227] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.758579 2530 ts_tablet_manager.cc:1144] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s I0521 22:30:05.760090 2530 tablet_bootstrap.cc:440] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92: Bootstrap starting. I0521 22:30:05.763464 2408 catalog_manager.cc:4202] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 reported cstate change: term changed from 0 to 1, leader changed from to 0910ecf09a21464eb0366a9258316f92 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "0910ecf09a21464eb0366a9258316f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } health_report { overall_health: HEALTHY } } } I0521 22:30:05.765489 2530 tablet_bootstrap.cc:596] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92: No blocks or log segments found. Creating new log. I0521 22:30:05.770967 2530 tablet_bootstrap.cc:440] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92: No bootstrap required, opened a new log I0521 22:30:05.771418 2530 ts_tablet_manager.cc:1115] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s I0521 22:30:05.773883 2530 raft_consensus.cc:342] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.774444 2530 raft_consensus.cc:368] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:05.774742 2530 raft_consensus.cc:722] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Initialized, Role: FOLLOWER I0521 22:30:05.775368 2530 consensus_queue.cc:250] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.776010 2530 raft_consensus.cc:382] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:05.776288 2530 raft_consensus.cc:478] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:05.776644 2530 raft_consensus.cc:2975] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:05.781846 2530 raft_consensus.cc:500] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.782466 2530 leader_election.cc:302] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0910ecf09a21464eb0366a9258316f92; no voters: I0521 22:30:05.782994 2530 leader_election.cc:288] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:05.783113 2532 raft_consensus.cc:2721] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:05.783507 2532 raft_consensus.cc:683] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Becoming Leader. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Running, Role: LEADER I0521 22:30:05.784108 2532 consensus_queue.cc:227] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.785159 2530 ts_tablet_manager.cc:1144] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s I0521 22:30:05.785902 2530 tablet_bootstrap.cc:440] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92: Bootstrap starting. I0521 22:30:05.789613 2408 catalog_manager.cc:4202] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 reported cstate change: term changed from 0 to 1, leader changed from to 0910ecf09a21464eb0366a9258316f92 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "0910ecf09a21464eb0366a9258316f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } health_report { overall_health: HEALTHY } } } I0521 22:30:05.790622 2530 tablet_bootstrap.cc:596] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92: No blocks or log segments found. Creating new log. I0521 22:30:05.806386 2530 tablet_bootstrap.cc:440] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92: No bootstrap required, opened a new log I0521 22:30:05.806730 2530 ts_tablet_manager.cc:1115] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.009s I0521 22:30:05.809111 2530 raft_consensus.cc:342] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.809473 2530 raft_consensus.cc:368] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:05.809684 2530 raft_consensus.cc:722] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Initialized, Role: FOLLOWER I0521 22:30:05.810111 2530 consensus_queue.cc:250] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.810497 2530 raft_consensus.cc:382] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:05.810673 2530 raft_consensus.cc:478] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:05.810881 2530 raft_consensus.cc:2975] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:05.813892 2530 raft_consensus.cc:500] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.814325 2530 leader_election.cc:302] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0910ecf09a21464eb0366a9258316f92; no voters: I0521 22:30:05.814685 2530 leader_election.cc:288] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:05.814844 2532 raft_consensus.cc:2721] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:05.815245 2532 raft_consensus.cc:683] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Becoming Leader. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Running, Role: LEADER I0521 22:30:05.815752 2530 ts_tablet_manager.cc:1144] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s I0521 22:30:05.816256 2530 tablet_bootstrap.cc:440] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92: Bootstrap starting. I0521 22:30:05.815824 2532 consensus_queue.cc:227] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.819548 2530 tablet_bootstrap.cc:596] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92: No blocks or log segments found. Creating new log. I0521 22:30:05.824723 2408 catalog_manager.cc:4202] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 reported cstate change: term changed from 0 to 1, leader changed from to 0910ecf09a21464eb0366a9258316f92 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "0910ecf09a21464eb0366a9258316f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } health_report { overall_health: HEALTHY } } } I0521 22:30:05.825706 2530 tablet_bootstrap.cc:440] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92: No bootstrap required, opened a new log I0521 22:30:05.825995 2530 ts_tablet_manager.cc:1115] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s I0521 22:30:05.827531 2530 raft_consensus.cc:342] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.827857 2530 raft_consensus.cc:368] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:05.828048 2530 raft_consensus.cc:722] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Initialized, Role: FOLLOWER I0521 22:30:05.828501 2530 consensus_queue.cc:250] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.828897 2530 raft_consensus.cc:382] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:05.829071 2530 raft_consensus.cc:478] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:05.829282 2530 raft_consensus.cc:2975] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:05.832379 2530 raft_consensus.cc:500] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.832788 2530 leader_election.cc:302] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0910ecf09a21464eb0366a9258316f92; no voters: I0521 22:30:05.833151 2530 leader_election.cc:288] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:05.833287 2532 raft_consensus.cc:2721] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:05.833983 2532 raft_consensus.cc:683] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Becoming Leader. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Running, Role: LEADER I0521 22:30:05.835291 2530 ts_tablet_manager.cc:1144] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s I0521 22:30:05.834985 2532 consensus_queue.cc:227] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.836107 2530 tablet_bootstrap.cc:440] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92: Bootstrap starting. I0521 22:30:05.841203 2530 tablet_bootstrap.cc:596] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92: No blocks or log segments found. Creating new log. I0521 22:30:05.844218 2408 catalog_manager.cc:4202] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 reported cstate change: term changed from 0 to 1, leader changed from to 0910ecf09a21464eb0366a9258316f92 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "0910ecf09a21464eb0366a9258316f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } health_report { overall_health: HEALTHY } } } I0521 22:30:05.846611 2530 tablet_bootstrap.cc:440] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92: No bootstrap required, opened a new log I0521 22:30:05.846922 2530 ts_tablet_manager.cc:1115] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s I0521 22:30:05.849068 2530 raft_consensus.cc:342] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.849584 2530 raft_consensus.cc:368] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:05.849859 2530 raft_consensus.cc:722] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Initialized, Role: FOLLOWER I0521 22:30:05.850544 2530 consensus_queue.cc:250] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.851106 2530 raft_consensus.cc:382] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:05.851298 2530 raft_consensus.cc:478] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:05.851538 2530 raft_consensus.cc:2975] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:05.855159 2530 raft_consensus.cc:500] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.855639 2530 leader_election.cc:302] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0910ecf09a21464eb0366a9258316f92; no voters: I0521 22:30:05.856083 2530 leader_election.cc:288] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:05.856215 2532 raft_consensus.cc:2721] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:05.856923 2532 raft_consensus.cc:683] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Becoming Leader. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Running, Role: LEADER I0521 22:30:05.857601 2530 ts_tablet_manager.cc:1144] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s I0521 22:30:05.857522 2532 consensus_queue.cc:227] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.858280 2530 tablet_bootstrap.cc:440] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92: Bootstrap starting. I0521 22:30:05.863596 2407 catalog_manager.cc:4202] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 reported cstate change: term changed from 0 to 1, leader changed from to 0910ecf09a21464eb0366a9258316f92 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "0910ecf09a21464eb0366a9258316f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } health_report { overall_health: HEALTHY } } } I0521 22:30:05.867161 2530 tablet_bootstrap.cc:596] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92: No blocks or log segments found. Creating new log. I0521 22:30:05.871395 2530 tablet_bootstrap.cc:440] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92: No bootstrap required, opened a new log I0521 22:30:05.871735 2530 ts_tablet_manager.cc:1115] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.002s I0521 22:30:05.873454 2530 raft_consensus.cc:342] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.873790 2530 raft_consensus.cc:368] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:05.873958 2530 raft_consensus.cc:722] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Initialized, Role: FOLLOWER I0521 22:30:05.874433 2530 consensus_queue.cc:250] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.874859 2530 raft_consensus.cc:382] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:05.875031 2530 raft_consensus.cc:478] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:05.875283 2530 raft_consensus.cc:2975] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:05.878268 2530 raft_consensus.cc:500] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.878679 2530 leader_election.cc:302] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0910ecf09a21464eb0366a9258316f92; no voters: I0521 22:30:05.879153 2530 leader_election.cc:288] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:05.879297 2532 raft_consensus.cc:2721] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:05.879884 2532 raft_consensus.cc:683] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Becoming Leader. State: Replica: 0910ecf09a21464eb0366a9258316f92, State: Running, Role: LEADER I0521 22:30:05.880352 2530 ts_tablet_manager.cc:1144] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s I0521 22:30:05.880494 2532 consensus_queue.cc:227] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } } I0521 22:30:05.886193 2407 catalog_manager.cc:4202] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 reported cstate change: term changed from 0 to 1, leader changed from to 0910ecf09a21464eb0366a9258316f92 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "0910ecf09a21464eb0366a9258316f92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0910ecf09a21464eb0366a9258316f92" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 36477 } health_report { overall_health: HEALTHY } } } I0521 22:30:05.934867 359 tablet_server.cc:157] TabletServer@127.0.89.193:0 shutting down... I0521 22:30:05.955356 359 ts_tablet_manager.cc:1199] Shutting down tablet manager... I0521 22:30:05.955910 359 tablet_replica.cc:277] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92: stopping tablet replica I0521 22:30:05.956547 359 raft_consensus.cc:2186] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Raft consensus shutting down. I0521 22:30:05.956872 359 raft_consensus.cc:2213] T d6e9964553ec42c6aa013b2fa01417a5 P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:30:05.959492 359 tablet_replica.cc:277] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92: stopping tablet replica I0521 22:30:05.959959 359 raft_consensus.cc:2186] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Raft consensus shutting down. I0521 22:30:05.960279 359 raft_consensus.cc:2213] T a894f47d27674d978d208a696539c513 P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:30:05.962478 359 tablet_replica.cc:277] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92: stopping tablet replica I0521 22:30:05.962899 359 raft_consensus.cc:2186] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Raft consensus shutting down. I0521 22:30:05.963160 359 raft_consensus.cc:2213] T 0e54106c9f2b41ddbd5598e955f7f2b9 P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:30:05.965147 359 tablet_replica.cc:277] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92: stopping tablet replica I0521 22:30:05.965551 359 raft_consensus.cc:2186] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Raft consensus shutting down. I0521 22:30:05.965808 359 raft_consensus.cc:2213] T a29148539a414d429508dcb4d038f2bc P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:30:05.967653 359 tablet_replica.cc:277] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92: stopping tablet replica I0521 22:30:05.968088 359 raft_consensus.cc:2186] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Raft consensus shutting down. I0521 22:30:05.968349 359 raft_consensus.cc:2213] T b46d06fd7afc4a64aca392c75116e7ac P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:30:05.970758 359 tablet_replica.cc:277] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92: stopping tablet replica I0521 22:30:05.971186 359 raft_consensus.cc:2186] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Raft consensus shutting down. I0521 22:30:05.971458 359 raft_consensus.cc:2213] T 35d79bbc87724c8a8e1d7617727056bf P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:30:05.973563 359 tablet_replica.cc:277] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92: stopping tablet replica I0521 22:30:05.973950 359 raft_consensus.cc:2186] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Raft consensus shutting down. I0521 22:30:05.974231 359 raft_consensus.cc:2213] T 550f3367b26d46b892a6c07eb4981bca P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:30:05.976835 359 tablet_replica.cc:277] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92: stopping tablet replica I0521 22:30:05.977298 359 raft_consensus.cc:2186] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Raft consensus shutting down. I0521 22:30:05.977579 359 raft_consensus.cc:2213] T 4199b9e1b41947a8bde6f17bd6245deb P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:30:05.980262 359 tablet_replica.cc:277] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92: stopping tablet replica I0521 22:30:05.980646 359 raft_consensus.cc:2186] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Raft consensus shutting down. I0521 22:30:05.980921 359 raft_consensus.cc:2213] T 0c6a8f75ffd44c92935ce8c24bb92a8e P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:30:05.983238 359 tablet_replica.cc:277] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92: stopping tablet replica I0521 22:30:05.983608 359 raft_consensus.cc:2186] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [term 1 LEADER]: Raft consensus shutting down. I0521 22:30:05.983880 359 raft_consensus.cc:2213] T 98f91e535b574584b4c783a29cdeef7d P 0910ecf09a21464eb0366a9258316f92 [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:30:06.071425 359 tablet_server.cc:171] TabletServer@127.0.89.193:0 shutdown complete. I0521 22:30:07.467303 359 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:30:07.482436 359 hybrid_clock.cc:243] auto-selected time source: system_unsync W0521 22:30:07.482595 359 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters. I0521 22:30:07.482751 359 hybrid_clock.cc:590] HybridClock initialized: now 1590100207482734 us; error 0 us; skew 500 ppm I0521 22:30:07.483827 359 fs_manager.cc:259] Metadata directory not provided I0521 22:30:07.483986 359 fs_manager.cc:262] Using existing metadata directory in first data directory I0521 22:30:07.499121 359 fs_manager.cc:410] Time spent opening directory manager: real 0.014s user 0.011s sys 0.004s I0521 22:30:07.506934 359 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:30:07.507091 359 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/ts-0-root uuid: "0910ecf09a21464eb0366a9258316f92" format_stamp: "Formatted at 2020-05-21 22:30:05 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:07.507390 359 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestLocalReplicaDeleteMultiple.1590100104622499-359/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0521 22:30:07.522382 359 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:30:07.522524 359 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:30:07.560695 359 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.193:36477 I0521 22:30:07.563088 359 webserver.cc:381] Webserver started at http://127.0.89.193:41991/ using document root and password file I0521 22:30:07.572273 359 tablet_server.cc:157] TabletServer@127.0.89.193:36477 shutting down... I0521 22:30:07.612466 2626 heartbeater.cc:325] Connected to a master server at 127.0.89.254:44861 I0521 22:30:07.614408 2626 heartbeater.cc:416] Registering TS with master... I0521 22:30:07.615360 2626 heartbeater.cc:465] Master 127.0.89.254:44861 requested a full tablet report, sending... I0521 22:30:07.617491 2407 ts_manager.cc:191] Re-registered known tserver with Master: 0910ecf09a21464eb0366a9258316f92 (127.0.89.193:36477) I0521 22:30:07.618955 2407 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43620 I0521 22:30:07.621395 359 ts_tablet_manager.cc:1199] Shutting down tablet manager... I0521 22:30:07.636183 359 tablet_server.cc:171] TabletServer@127.0.89.193:36477 shutdown complete. I0521 22:30:07.641595 359 master.cc:274] Master@127.0.89.254:44861 shutting down... I0521 22:30:07.664914 359 raft_consensus.cc:2186] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [term 1 LEADER]: Raft consensus shutting down. I0521 22:30:07.665269 359 raft_consensus.cc:2213] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143 [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:30:07.665536 359 tablet_replica.cc:277] T 00000000000000000000000000000000 P 47ff8aad212b4c20850c8579a322c143: stopping tablet replica I0521 22:30:07.682260 359 master.cc:288] Master@127.0.89.254:44861 shutdown complete. [ OK ] ToolTest.TestLocalReplicaDeleteMultiple (4289 ms) [ RUN ] ToolTest.TestTserverListLocationAssigned 2020-05-21T22:30:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-05-21T22:30:07Z Disabled control of system clock I0521 22:30:07.736371 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --ipki_ca_key_size=768 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:43899 --webserver_interface=127.0.89.254 --webserver_port=0 --builtin_ntp_servers=127.0.89.212:38107 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true --location_mapping_cmd=/tmp/dist-test-taskopyat3/build/tsan/bin/testdata/first_argument.sh /foo-bar0/BAAZ._9-quux with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:30:07.896545 2633 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:30:07.897049 2633 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:30:07.897506 2633 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:30:07.909783 2633 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=768 W0521 22:30:07.909958 2633 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:30:07.910168 2633 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0521 22:30:07.910362 2633 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0521 22:30:07.929497 2633 master_runner.cc:93] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:38107 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/wal --location_mapping_cmd=/tmp/dist-test-taskopyat3/build/tsan/bin/testdata/first_argument.sh /foo-bar0/BAAZ._9-quux --ipki_ca_key_size=768 --ipki_server_key_size=768 --openssl_security_level_override=0 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:43899 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.89.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:30:07.930461 2633 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:30:07.932010 2633 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:30:07.947574 2633 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:30:07.950551 2633 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:30:07.951885 2633 hybrid_clock.cc:590] HybridClock initialized: now 1590100207951829 us; error 64 us; skew 500 ppm I0521 22:30:07.953650 2633 fs_manager.cc:259] Metadata directory not provided I0521 22:30:07.953821 2633 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:30:07.954195 2633 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:30:07.957502 2633 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/data/instance: uuid: "b02d64ede5934268a21f8db4b19d6469" format_stamp: "Formatted at 2020-05-21 22:30:07 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:07.958398 2633 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/wal/instance: uuid: "b02d64ede5934268a21f8db4b19d6469" format_stamp: "Formatted at 2020-05-21 22:30:07 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:07.978498 2633 fs_manager.cc:518] Time spent creating directory manager: real 0.020s user 0.018s sys 0.003s I0521 22:30:07.986840 2633 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:30:07.987002 2633 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/wal uuid: "b02d64ede5934268a21f8db4b19d6469" format_stamp: "Formatted at 2020-05-21 22:30:07 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:07.987308 2633 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:30:08.739787 2633 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.722s user 0.297s sys 0.424s W0521 22:30:08.740011 2633 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.722s user 0.297s sys 0.424s I0521 22:30:08.761279 2633 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:30:08.833689 2633 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.254:43899 I0521 22:30:08.838850 2633 webserver.cc:381] Webserver started at http://127.0.89.254:45191/ using document root and password file I0521 22:30:08.840402 2633 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/master-0/data/info.pb I0521 22:30:08.845266 2695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:08.848632 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master as pid 2633 I0521 22:30:08.850991 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --rpc_bind_addresses=127.0.89.193:0 --local_ip_for_outbound_sockets=127.0.89.193 --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:43899 --builtin_ntp_servers=127.0.89.212:38107 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} I0521 22:30:08.870362 2695 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469: Bootstrap starting. I0521 22:30:08.877146 2695 tablet_bootstrap.cc:596] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469: No blocks or log segments found. Creating new log. I0521 22:30:08.879408 2695 log.cc:819] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469: Log is configured to *not* fsync() on all Append() calls I0521 22:30:08.886530 2695 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469: No bootstrap required, opened a new log I0521 22:30:08.901701 2695 raft_consensus.cc:342] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d64ede5934268a21f8db4b19d6469" member_type: VOTER } I0521 22:30:08.902086 2695 raft_consensus.cc:368] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:08.902290 2695 raft_consensus.cc:722] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02d64ede5934268a21f8db4b19d6469, State: Initialized, Role: FOLLOWER I0521 22:30:08.902887 2695 consensus_queue.cc:250] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d64ede5934268a21f8db4b19d6469" member_type: VOTER } I0521 22:30:08.903290 2695 raft_consensus.cc:382] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:08.903519 2695 raft_consensus.cc:478] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:08.903764 2695 raft_consensus.cc:2975] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:08.907055 2695 raft_consensus.cc:500] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d64ede5934268a21f8db4b19d6469" member_type: VOTER } I0521 22:30:08.907574 2695 leader_election.cc:302] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b02d64ede5934268a21f8db4b19d6469; no voters: I0521 22:30:08.909301 2695 leader_election.cc:288] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [CANDIDATE]: Term 1 election: Requested vote from peers WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:30:09.040381 2696 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:30:09.040979 2696 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:30:09.041772 2696 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:30:09.061091 2696 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:30:09.062574 2696 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.89.193 I0521 22:30:09.081877 2696 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:38107 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/wal --ipki_server_key_size=768 --openssl_security_level_override=0 --rpc_bind_addresses=127.0.89.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:43899 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.89.193 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:30:09.082859 2696 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:30:09.084566 2696 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:30:09.101610 2696 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:30:08.909682 2698 raft_consensus.cc:2721] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:09.104435 2698 raft_consensus.cc:683] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [term 1 LEADER]: Becoming Leader. State: Replica: b02d64ede5934268a21f8db4b19d6469, State: Running, Role: LEADER I0521 22:30:09.105182 2698 consensus_queue.cc:227] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d64ede5934268a21f8db4b19d6469" member_type: VOTER } I0521 22:30:09.106503 2696 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:30:09.107898 2696 hybrid_clock.cc:590] HybridClock initialized: now 1590100209107869 us; error 43 us; skew 500 ppm I0521 22:30:09.109359 2695 sys_catalog.cc:458] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [sys.catalog]: configured and running, proceeding with master startup. I0521 22:30:09.111348 2696 fs_manager.cc:259] Metadata directory not provided I0521 22:30:09.111605 2696 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:30:09.111435 2699 sys_catalog.cc:357] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b02d64ede5934268a21f8db4b19d6469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d64ede5934268a21f8db4b19d6469" member_type: VOTER } } I0521 22:30:09.112120 2696 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:30:09.112231 2699 sys_catalog.cc:360] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [sys.catalog]: This master's current role is: LEADER I0521 22:30:09.117183 2696 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/data/instance: uuid: "7ef34b05398a4de280844b662ae8e26a" format_stamp: "Formatted at 2020-05-21 22:30:09 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:09.116777 2705 sys_catalog.cc:357] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b02d64ede5934268a21f8db4b19d6469. Latest consensus state: current_term: 1 leader_uuid: "b02d64ede5934268a21f8db4b19d6469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02d64ede5934268a21f8db4b19d6469" member_type: VOTER } } I0521 22:30:09.119179 2705 sys_catalog.cc:360] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469 [sys.catalog]: This master's current role is: LEADER I0521 22:30:09.119355 2696 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/wal/instance: uuid: "7ef34b05398a4de280844b662ae8e26a" format_stamp: "Formatted at 2020-05-21 22:30:09 on dist-test-slave-dist-test-slave-jp1g" W0521 22:30:09.130138 2709 catalog_manager.cc:1201] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0521 22:30:09.130410 2709 catalog_manager.cc:667] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0521 22:30:09.130769 2710 catalog_manager.cc:1131] Loading table and tablet metadata into memory... I0521 22:30:09.134740 2710 catalog_manager.cc:1142] Initializing Kudu internal certificate authority... I0521 22:30:09.157610 2696 fs_manager.cc:518] Time spent creating directory manager: real 0.038s user 0.034s sys 0.003s I0521 22:30:09.170625 2710 catalog_manager.cc:1026] Generated new certificate authority record I0521 22:30:09.172255 2710 catalog_manager.cc:1151] Loading token signing keys... I0521 22:30:09.175513 2696 fs_manager.cc:433] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0521 22:30:09.175731 2696 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/wal uuid: "7ef34b05398a4de280844b662ae8e26a" format_stamp: "Formatted at 2020-05-21 22:30:09 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:09.176158 2696 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0521 22:30:09.185111 2710 catalog_manager.cc:4538] T 00000000000000000000000000000000 P b02d64ede5934268a21f8db4b19d6469: Generated new TSK 0 I0521 22:30:09.185750 2710 catalog_manager.cc:1161] Initializing in-progress tserver states... W0521 22:30:10.261356 2696 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 1.045s user 0.000s sys 0.002s W0521 22:30:10.261600 2696 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 1.045s user 0.000s sys 0.002s I0521 22:30:10.345898 2696 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:30:10.350653 2696 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:30:10.350822 2696 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:30:10.752930 2696 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.193:34781 I0521 22:30:10.756332 2696 webserver.cc:381] Webserver started at http://127.0.89.193:45481/ using document root and password file I0521 22:30:10.758955 2696 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestTserverListLocationAssigned.1590100104622499-359/minicluster-data/ts-0/data/info.pb I0521 22:30:10.759848 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver as pid 2696 I0521 22:30:10.804167 2828 heartbeater.cc:325] Connected to a master server at 127.0.89.254:43899 I0521 22:30:10.804538 2828 heartbeater.cc:416] Registering TS with master... I0521 22:30:10.805227 2828 heartbeater.cc:465] Master 127.0.89.254:43899 requested a full tablet report, sending... I0521 22:30:10.825301 2662 ts_manager.cc:191] Registered new tserver with Master: 7ef34b05398a4de280844b662ae8e26a (127.0.89.193:34781) I0521 22:30:10.826419 359 external_mini_cluster.cc:699] 1 TS(s) registered with all masters I0521 22:30:10.828698 2662 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.89.193:33333 I0521 22:30:11.833451 2828 heartbeater.cc:457] Master 127.0.89.254:43899 was elected leader, sending a full tablet report... I0521 22:30:13.171308 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver with pid 2696 I0521 22:30:13.185276 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master with pid 2633 2020-05-21T22:30:13Z chronyd exiting [ OK ] ToolTest.TestTserverListLocationAssigned (5523 ms) [ RUN ] ToolTest.TestDeleteTable 2020-05-21T22:30:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-05-21T22:30:13Z Disabled control of system clock I0521 22:30:13.257177 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --ipki_ca_key_size=768 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:45991 --webserver_interface=127.0.89.254 --webserver_port=0 --builtin_ntp_servers=127.0.89.212:42329 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:30:13.419747 2852 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:30:13.420339 2852 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:30:13.420856 2852 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:30:13.433409 2852 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=768 W0521 22:30:13.433619 2852 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:30:13.433881 2852 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0521 22:30:13.434118 2852 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0521 22:30:13.453307 2852 master_runner.cc:93] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:42329 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/wal --ipki_ca_key_size=768 --ipki_server_key_size=768 --openssl_security_level_override=0 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:45991 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.89.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:30:13.454205 2852 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:30:13.455713 2852 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:30:13.470597 2852 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:30:13.473891 2852 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:30:13.475248 2852 hybrid_clock.cc:590] HybridClock initialized: now 1590100213475221 us; error 31 us; skew 500 ppm I0521 22:30:13.477284 2852 fs_manager.cc:259] Metadata directory not provided I0521 22:30:13.477428 2852 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:30:13.477742 2852 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:30:13.480856 2852 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/data/instance: uuid: "fd1a523e191a4cf9a15db0fb739a9d4e" format_stamp: "Formatted at 2020-05-21 22:30:13 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:13.481637 2852 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/wal/instance: uuid: "fd1a523e191a4cf9a15db0fb739a9d4e" format_stamp: "Formatted at 2020-05-21 22:30:13 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:13.501838 2852 fs_manager.cc:518] Time spent creating directory manager: real 0.020s user 0.014s sys 0.007s I0521 22:30:13.509975 2852 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:30:13.510113 2852 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/wal uuid: "fd1a523e191a4cf9a15db0fb739a9d4e" format_stamp: "Formatted at 2020-05-21 22:30:13 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:13.510371 2852 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:30:14.337517 2852 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.801s user 0.286s sys 0.495s W0521 22:30:14.337818 2852 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.801s user 0.286s sys 0.495s I0521 22:30:14.362746 2852 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:30:14.431687 2852 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.254:45991 I0521 22:30:14.435268 2852 webserver.cc:381] Webserver started at http://127.0.89.254:33765/ using document root and password file I0521 22:30:14.436262 2852 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/master-0/data/info.pb I0521 22:30:14.438817 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master as pid 2852 I0521 22:30:14.441284 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --rpc_bind_addresses=127.0.89.193:0 --local_ip_for_outbound_sockets=127.0.89.193 --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:45991 --builtin_ntp_servers=127.0.89.212:42329 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} I0521 22:30:14.442124 2914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:14.471680 2914 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e: Bootstrap starting. I0521 22:30:14.480635 2914 tablet_bootstrap.cc:596] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e: No blocks or log segments found. Creating new log. I0521 22:30:14.482692 2914 log.cc:819] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e: Log is configured to *not* fsync() on all Append() calls I0521 22:30:14.486876 2914 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e: No bootstrap required, opened a new log I0521 22:30:14.502001 2914 raft_consensus.cc:342] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1a523e191a4cf9a15db0fb739a9d4e" member_type: VOTER } I0521 22:30:14.502387 2914 raft_consensus.cc:368] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:14.502588 2914 raft_consensus.cc:722] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1a523e191a4cf9a15db0fb739a9d4e, State: Initialized, Role: FOLLOWER I0521 22:30:14.503197 2914 consensus_queue.cc:250] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1a523e191a4cf9a15db0fb739a9d4e" member_type: VOTER } I0521 22:30:14.503585 2914 raft_consensus.cc:382] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:14.503819 2914 raft_consensus.cc:478] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:14.504088 2914 raft_consensus.cc:2975] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:14.507287 2914 raft_consensus.cc:500] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1a523e191a4cf9a15db0fb739a9d4e" member_type: VOTER } I0521 22:30:14.507817 2914 leader_election.cc:302] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd1a523e191a4cf9a15db0fb739a9d4e; no voters: I0521 22:30:14.509426 2917 raft_consensus.cc:2721] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [term 1 FOLLOWER]: Leader election won for term 1 WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:30:14.645998 2915 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:30:14.646662 2915 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:30:14.647509 2915 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:30:14.666838 2915 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:30:14.668361 2915 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.89.193 I0521 22:30:14.510490 2914 leader_election.cc:288] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:14.694159 2917 raft_consensus.cc:683] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [term 1 LEADER]: Becoming Leader. State: Replica: fd1a523e191a4cf9a15db0fb739a9d4e, State: Running, Role: LEADER I0521 22:30:14.695145 2917 consensus_queue.cc:227] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1a523e191a4cf9a15db0fb739a9d4e" member_type: VOTER } I0521 22:30:14.696899 2915 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:42329 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/wal --ipki_server_key_size=768 --openssl_security_level_override=0 --rpc_bind_addresses=127.0.89.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:45991 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.89.193 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:30:14.698108 2915 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:30:14.699715 2915 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:30:14.701752 2918 sys_catalog.cc:357] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd1a523e191a4cf9a15db0fb739a9d4e. Latest consensus state: current_term: 1 leader_uuid: "fd1a523e191a4cf9a15db0fb739a9d4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1a523e191a4cf9a15db0fb739a9d4e" member_type: VOTER } } I0521 22:30:14.702271 2918 sys_catalog.cc:360] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [sys.catalog]: This master's current role is: LEADER I0521 22:30:14.704133 2914 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [sys.catalog]: configured and running, proceeding with master startup. I0521 22:30:14.703586 2923 sys_catalog.cc:357] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd1a523e191a4cf9a15db0fb739a9d4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1a523e191a4cf9a15db0fb739a9d4e" member_type: VOTER } } I0521 22:30:14.706576 2923 sys_catalog.cc:360] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e [sys.catalog]: This master's current role is: LEADER I0521 22:30:14.707263 2925 catalog_manager.cc:1131] Loading table and tablet metadata into memory... I0521 22:30:14.714161 2925 catalog_manager.cc:1142] Initializing Kudu internal certificate authority... I0521 22:30:14.726990 2915 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:30:14.730487 2915 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:30:14.731901 2915 hybrid_clock.cc:590] HybridClock initialized: now 1590100214731841 us; error 68 us; skew 500 ppm I0521 22:30:14.734169 2915 fs_manager.cc:259] Metadata directory not provided I0521 22:30:14.734374 2915 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:30:14.734863 2915 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:30:14.739699 2915 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/data/instance: uuid: "115f275699404963944f19bbb793a77b" format_stamp: "Formatted at 2020-05-21 22:30:14 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:14.741205 2915 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/wal/instance: uuid: "115f275699404963944f19bbb793a77b" format_stamp: "Formatted at 2020-05-21 22:30:14 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:14.742796 2925 catalog_manager.cc:1026] Generated new certificate authority record I0521 22:30:14.744827 2925 catalog_manager.cc:1151] Loading token signing keys... I0521 22:30:14.754086 2925 catalog_manager.cc:4538] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e: Generated new TSK 0 I0521 22:30:14.754751 2925 catalog_manager.cc:1161] Initializing in-progress tserver states... I0521 22:30:14.778334 2915 fs_manager.cc:518] Time spent creating directory manager: real 0.036s user 0.036s sys 0.000s I0521 22:30:14.792922 2915 fs_manager.cc:433] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0521 22:30:14.793146 2915 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/wal uuid: "115f275699404963944f19bbb793a77b" format_stamp: "Formatted at 2020-05-21 22:30:14 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:14.793552 2915 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:30:15.817417 2915 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.977s user 0.000s sys 0.002s W0521 22:30:15.817662 2915 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.977s user 0.000s sys 0.002s I0521 22:30:15.858721 2915 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:30:15.865227 2915 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:30:15.865451 2915 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:30:16.054909 2915 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.193:46807 I0521 22:30:16.058223 2915 webserver.cc:381] Webserver started at http://127.0.89.193:43197/ using document root and password file I0521 22:30:16.059093 2915 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/data/info.pb I0521 22:30:16.066459 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver as pid 2915 I0521 22:30:16.091868 3048 heartbeater.cc:325] Connected to a master server at 127.0.89.254:45991 I0521 22:30:16.092264 3048 heartbeater.cc:416] Registering TS with master... I0521 22:30:16.092936 3048 heartbeater.cc:465] Master 127.0.89.254:45991 requested a full tablet report, sending... I0521 22:30:16.095257 2880 ts_manager.cc:191] Registered new tserver with Master: 115f275699404963944f19bbb793a77b (127.0.89.193:46807) I0521 22:30:16.098222 2880 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.89.193:32941 I0521 22:30:16.098362 359 external_mini_cluster.cc:699] 1 TS(s) registered with all masters I0521 22:30:16.137306 359 test_util.cc:232] Using random seed: 243942978 I0521 22:30:16.192555 2880 catalog_manager.cc:1501] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57468: name: "kudu.table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0521 22:30:16.251844 2985 tablet_service.cc:1232] Processing CreateTablet for tablet 582f86a7e1154214a0463ccc38c4b0fa (table=kudu.table [id=62356851b1704dab8d6bc84902ee9557]), partition=RANGE (key) PARTITION UNBOUNDED I0521 22:30:16.253386 2985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582f86a7e1154214a0463ccc38c4b0fa. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:16.281137 3070 tablet_bootstrap.cc:440] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b: Bootstrap starting. I0521 22:30:16.289666 3070 tablet_bootstrap.cc:596] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b: No blocks or log segments found. Creating new log. I0521 22:30:16.291175 3070 log.cc:819] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b: Log is configured to *not* fsync() on all Append() calls I0521 22:30:16.299363 3070 tablet_bootstrap.cc:440] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b: No bootstrap required, opened a new log I0521 22:30:16.299793 3070 ts_tablet_manager.cc:1115] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s I0521 22:30:16.321415 3070 raft_consensus.cc:342] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115f275699404963944f19bbb793a77b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 46807 } } I0521 22:30:16.322162 3070 raft_consensus.cc:368] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:16.322417 3070 raft_consensus.cc:722] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 115f275699404963944f19bbb793a77b, State: Initialized, Role: FOLLOWER I0521 22:30:16.323146 3070 consensus_queue.cc:250] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115f275699404963944f19bbb793a77b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 46807 } } I0521 22:30:16.323804 3070 raft_consensus.cc:382] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:16.324164 3070 raft_consensus.cc:478] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:16.324497 3070 raft_consensus.cc:2975] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:16.329648 3070 raft_consensus.cc:500] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115f275699404963944f19bbb793a77b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 46807 } } I0521 22:30:16.330407 3070 leader_election.cc:302] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 115f275699404963944f19bbb793a77b; no voters: I0521 22:30:16.332504 3070 leader_election.cc:288] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:16.332880 3072 raft_consensus.cc:2721] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:16.453871 3048 heartbeater.cc:457] Master 127.0.89.254:45991 was elected leader, sending a full tablet report... I0521 22:30:16.456455 3072 raft_consensus.cc:683] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [term 1 LEADER]: Becoming Leader. State: Replica: 115f275699404963944f19bbb793a77b, State: Running, Role: LEADER I0521 22:30:16.457235 3072 consensus_queue.cc:227] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115f275699404963944f19bbb793a77b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 46807 } } I0521 22:30:16.460512 3070 ts_tablet_manager.cc:1144] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b: Time spent starting tablet: real 0.160s user 0.094s sys 0.060s I0521 22:30:16.473104 2880 catalog_manager.cc:4202] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b reported cstate change: term changed from 0 to 1, leader changed from to 115f275699404963944f19bbb793a77b (127.0.89.193). New cstate: current_term: 1 leader_uuid: "115f275699404963944f19bbb793a77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115f275699404963944f19bbb793a77b" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 46807 } health_report { overall_health: HEALTHY } } } W0521 22:30:16.478922 2866 debug-util.cc:402] Leaking SignalData structure 0x7b080002b380 after lost signal to thread 2853 I0521 22:30:17.887962 2880 catalog_manager.cc:1999] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57472: table { table_name: "kudu.table" } modify_external_catalogs: false I0521 22:30:17.901546 2880 catalog_manager.cc:4473] T 00000000000000000000000000000000 P fd1a523e191a4cf9a15db0fb739a9d4e: Sending DeleteTablet for 1 replicas of tablet 582f86a7e1154214a0463ccc38c4b0fa I0521 22:30:17.904114 2985 tablet_service.cc:1276] Processing DeleteTablet for tablet 582f86a7e1154214a0463ccc38c4b0fa with delete_type TABLET_DATA_DELETED (Table deleted at 2020-05-21 22:30:17 UTC) from {username='slave'} at 127.0.0.1:54944 I0521 22:30:17.906399 3094 tablet_replica.cc:277] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b: stopping tablet replica I0521 22:30:17.907110 3094 raft_consensus.cc:2186] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [term 1 LEADER]: Raft consensus shutting down. I0521 22:30:17.907590 3094 raft_consensus.cc:2213] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b [term 1 FOLLOWER]: Raft consensus is shut down! I0521 22:30:17.910109 3094 ts_tablet_manager.cc:1476] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b: Deleting tablet data with delete state TABLET_DATA_DELETED I0521 22:30:17.921828 3094 ts_tablet_manager.cc:1489] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 I0521 22:30:17.922235 3094 log.cc:1189] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b: Deleting WAL directory at /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestDeleteTable.1590100104622499-359/minicluster-data/ts-0/wal/wals/582f86a7e1154214a0463ccc38c4b0fa I0521 22:30:17.923149 3094 ts_tablet_manager.cc:1510] T 582f86a7e1154214a0463ccc38c4b0fa P 115f275699404963944f19bbb793a77b: Deleting consensus metadata I0521 22:30:17.926513 2865 catalog_manager.cc:3571] TS 115f275699404963944f19bbb793a77b (127.0.89.193:46807): tablet 582f86a7e1154214a0463ccc38c4b0fa (table kudu.table [id=62356851b1704dab8d6bc84902ee9557]) successfully deleted I0521 22:30:18.936911 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver with pid 2915 I0521 22:30:18.954284 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master with pid 2852 2020-05-21T22:30:18Z chronyd exiting [ OK ] ToolTest.TestDeleteTable (5777 ms) [ RUN ] ToolTest.TestScanTablePredicates 2020-05-21T22:30:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-05-21T22:30:19Z Disabled control of system clock I0521 22:30:19.031525 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --ipki_ca_key_size=768 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:38185 --webserver_interface=127.0.89.254 --webserver_port=0 --builtin_ntp_servers=127.0.89.212:40049 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:30:19.188449 3098 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:30:19.189009 3098 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:30:19.189519 3098 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:30:19.201978 3098 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=768 W0521 22:30:19.202185 3098 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:30:19.202430 3098 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0521 22:30:19.202667 3098 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0521 22:30:19.221899 3098 master_runner.cc:93] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:40049 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/wal --ipki_ca_key_size=768 --ipki_server_key_size=768 --openssl_security_level_override=0 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:38185 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.89.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:30:19.222760 3098 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:30:19.224326 3098 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:30:19.239384 3098 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:30:19.243122 3098 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:30:19.244470 3098 hybrid_clock.cc:590] HybridClock initialized: now 1590100219244432 us; error 42 us; skew 500 ppm I0521 22:30:19.246536 3098 fs_manager.cc:259] Metadata directory not provided I0521 22:30:19.246683 3098 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:30:19.247011 3098 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:30:19.250164 3098 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/data/instance: uuid: "73f2192a0af74371b34ba98b282c9489" format_stamp: "Formatted at 2020-05-21 22:30:19 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:19.251008 3098 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/wal/instance: uuid: "73f2192a0af74371b34ba98b282c9489" format_stamp: "Formatted at 2020-05-21 22:30:19 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:19.270815 3098 fs_manager.cc:518] Time spent creating directory manager: real 0.019s user 0.022s sys 0.000s I0521 22:30:19.279034 3098 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:30:19.279189 3098 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/wal uuid: "73f2192a0af74371b34ba98b282c9489" format_stamp: "Formatted at 2020-05-21 22:30:19 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:19.279448 3098 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:30:20.026549 3098 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.720s user 0.308s sys 0.412s W0521 22:30:20.026867 3098 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.721s user 0.308s sys 0.412s I0521 22:30:20.049193 3098 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:30:20.126655 3098 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.254:38185 I0521 22:30:20.130632 3098 webserver.cc:381] Webserver started at http://127.0.89.254:41357/ using document root and password file I0521 22:30:20.131606 3098 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/master-0/data/info.pb I0521 22:30:20.136373 3160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:20.141638 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master as pid 3098 I0521 22:30:20.144065 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --rpc_bind_addresses=127.0.89.193:0 --local_ip_for_outbound_sockets=127.0.89.193 --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:38185 --builtin_ntp_servers=127.0.89.212:40049 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} I0521 22:30:20.154247 3160 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489: Bootstrap starting. I0521 22:30:20.163441 3160 tablet_bootstrap.cc:596] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489: No blocks or log segments found. Creating new log. I0521 22:30:20.165591 3160 log.cc:819] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489: Log is configured to *not* fsync() on all Append() calls I0521 22:30:20.171043 3160 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489: No bootstrap required, opened a new log I0521 22:30:20.195359 3160 raft_consensus.cc:342] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2192a0af74371b34ba98b282c9489" member_type: VOTER } I0521 22:30:20.195957 3160 raft_consensus.cc:368] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:20.196216 3160 raft_consensus.cc:722] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f2192a0af74371b34ba98b282c9489, State: Initialized, Role: FOLLOWER I0521 22:30:20.196959 3160 consensus_queue.cc:250] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2192a0af74371b34ba98b282c9489" member_type: VOTER } I0521 22:30:20.197500 3160 raft_consensus.cc:382] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:20.197788 3160 raft_consensus.cc:478] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:20.198107 3160 raft_consensus.cc:2975] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:20.203104 3160 raft_consensus.cc:500] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2192a0af74371b34ba98b282c9489" member_type: VOTER } I0521 22:30:20.203824 3160 leader_election.cc:302] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73f2192a0af74371b34ba98b282c9489; no voters: I0521 22:30:20.205730 3160 leader_election.cc:288] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:20.206079 3163 raft_consensus.cc:2721] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:20.328238 3163 raft_consensus.cc:683] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [term 1 LEADER]: Becoming Leader. State: Replica: 73f2192a0af74371b34ba98b282c9489, State: Running, Role: LEADER I0521 22:30:20.329102 3163 consensus_queue.cc:227] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2192a0af74371b34ba98b282c9489" member_type: VOTER } I0521 22:30:20.333591 3160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [sys.catalog]: configured and running, proceeding with master startup. I0521 22:30:20.335052 3165 sys_catalog.cc:357] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73f2192a0af74371b34ba98b282c9489. Latest consensus state: current_term: 1 leader_uuid: "73f2192a0af74371b34ba98b282c9489" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2192a0af74371b34ba98b282c9489" member_type: VOTER } } I0521 22:30:20.334075 3164 sys_catalog.cc:357] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73f2192a0af74371b34ba98b282c9489" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f2192a0af74371b34ba98b282c9489" member_type: VOTER } } I0521 22:30:20.338800 3164 sys_catalog.cc:360] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [sys.catalog]: This master's current role is: LEADER I0521 22:30:20.339380 3165 sys_catalog.cc:360] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489 [sys.catalog]: This master's current role is: LEADER W0521 22:30:20.347523 3167 catalog_manager.cc:1201] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0521 22:30:20.347728 3167 catalog_manager.cc:667] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0521 22:30:20.348124 3169 catalog_manager.cc:1131] Loading table and tablet metadata into memory... I0521 22:30:20.352376 3169 catalog_manager.cc:1142] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:30:20.372887 3161 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:30:20.373600 3161 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:30:20.374481 3161 flags.cc:403] Enabled unsafe flag: --never_fsync=true I0521 22:30:20.378738 3169 catalog_manager.cc:1026] Generated new certificate authority record I0521 22:30:20.380493 3169 catalog_manager.cc:1151] Loading token signing keys... W0521 22:30:20.393965 3161 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:30:20.395582 3161 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.89.193 I0521 22:30:20.406919 3169 catalog_manager.cc:4538] T 00000000000000000000000000000000 P 73f2192a0af74371b34ba98b282c9489: Generated new TSK 0 I0521 22:30:20.408116 3169 catalog_manager.cc:1161] Initializing in-progress tserver states... I0521 22:30:20.422209 3161 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:40049 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/wal --ipki_server_key_size=768 --openssl_security_level_override=0 --rpc_bind_addresses=127.0.89.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:38185 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.89.193 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:30:20.423135 3161 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:30:20.424703 3161 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:30:20.440944 3161 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:30:20.448355 3161 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:30:20.449656 3161 hybrid_clock.cc:590] HybridClock initialized: now 1590100220449624 us; error 55 us; skew 500 ppm I0521 22:30:20.452790 3161 fs_manager.cc:259] Metadata directory not provided I0521 22:30:20.452944 3161 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:30:20.453315 3161 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:30:20.456993 3161 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/data/instance: uuid: "a180b25a6f8740bfa74c5ecf0ce21d36" format_stamp: "Formatted at 2020-05-21 22:30:20 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:20.457854 3161 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/wal/instance: uuid: "a180b25a6f8740bfa74c5ecf0ce21d36" format_stamp: "Formatted at 2020-05-21 22:30:20 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:20.481384 3161 fs_manager.cc:518] Time spent creating directory manager: real 0.023s user 0.016s sys 0.008s I0521 22:30:20.489564 3161 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0521 22:30:20.489706 3161 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/wal uuid: "a180b25a6f8740bfa74c5ecf0ce21d36" format_stamp: "Formatted at 2020-05-21 22:30:20 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:20.490015 3161 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:30:21.612877 3161 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 1.083s user 0.002s sys 0.000s W0521 22:30:21.613169 3161 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 1.083s user 0.002s sys 0.000s I0521 22:30:21.649242 3161 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:30:21.655737 3161 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:30:21.655980 3161 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:30:21.952836 3161 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.193:39743 I0521 22:30:21.956068 3161 webserver.cc:381] Webserver started at http://127.0.89.193:34471/ using document root and password file I0521 22:30:21.956938 3161 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestScanTablePredicates.1590100104622499-359/minicluster-data/ts-0/data/info.pb I0521 22:30:21.966850 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver as pid 3161 I0521 22:30:21.988422 3293 heartbeater.cc:325] Connected to a master server at 127.0.89.254:38185 I0521 22:30:21.988788 3293 heartbeater.cc:416] Registering TS with master... I0521 22:30:21.989735 3293 heartbeater.cc:465] Master 127.0.89.254:38185 requested a full tablet report, sending... I0521 22:30:21.991988 3127 ts_manager.cc:191] Registered new tserver with Master: a180b25a6f8740bfa74c5ecf0ce21d36 (127.0.89.193:39743) I0521 22:30:21.993934 359 external_mini_cluster.cc:699] 1 TS(s) registered with all masters I0521 22:30:21.993983 3127 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.89.193:57075 I0521 22:30:21.997016 359 test_util.cc:232] Using random seed: 249802695 I0521 22:30:22.030097 3127 catalog_manager.cc:1501] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55786: name: "kudu.table.scan.predicates" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0521 22:30:22.070092 3230 tablet_service.cc:1232] Processing CreateTablet for tablet cd23dbaf55f7426a948169cb684595ce (table=kudu.table.scan.predicates [id=ef8766e66f9a42a4a94fe4817c8d31f5]), partition=RANGE (key) PARTITION UNBOUNDED I0521 22:30:22.071544 3230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd23dbaf55f7426a948169cb684595ce. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:22.095924 3310 tablet_bootstrap.cc:440] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36: Bootstrap starting. I0521 22:30:22.101162 3310 tablet_bootstrap.cc:596] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36: No blocks or log segments found. Creating new log. I0521 22:30:22.102677 3310 log.cc:819] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36: Log is configured to *not* fsync() on all Append() calls I0521 22:30:22.106402 3310 tablet_bootstrap.cc:440] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36: No bootstrap required, opened a new log I0521 22:30:22.106721 3310 ts_tablet_manager.cc:1115] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s I0521 22:30:22.120508 3310 raft_consensus.cc:342] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a180b25a6f8740bfa74c5ecf0ce21d36" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 39743 } } I0521 22:30:22.121001 3310 raft_consensus.cc:368] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:22.121215 3310 raft_consensus.cc:722] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a180b25a6f8740bfa74c5ecf0ce21d36, State: Initialized, Role: FOLLOWER I0521 22:30:22.121791 3310 consensus_queue.cc:250] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a180b25a6f8740bfa74c5ecf0ce21d36" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 39743 } } I0521 22:30:22.122234 3310 raft_consensus.cc:382] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:22.122449 3310 raft_consensus.cc:478] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:22.122685 3310 raft_consensus.cc:2975] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:22.126420 3310 raft_consensus.cc:500] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a180b25a6f8740bfa74c5ecf0ce21d36" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 39743 } } I0521 22:30:22.126951 3310 leader_election.cc:302] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a180b25a6f8740bfa74c5ecf0ce21d36; no voters: I0521 22:30:22.128441 3310 leader_election.cc:288] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:22.128830 3312 raft_consensus.cc:2721] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:22.343521 3293 heartbeater.cc:457] Master 127.0.89.254:38185 was elected leader, sending a full tablet report... I0521 22:30:22.344233 3310 ts_tablet_manager.cc:1144] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36: Time spent starting tablet: real 0.237s user 0.102s sys 0.137s I0521 22:30:22.346227 3312 raft_consensus.cc:683] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [term 1 LEADER]: Becoming Leader. State: Replica: a180b25a6f8740bfa74c5ecf0ce21d36, State: Running, Role: LEADER I0521 22:30:22.346990 3312 consensus_queue.cc:227] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a180b25a6f8740bfa74c5ecf0ce21d36" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 39743 } } I0521 22:30:22.354338 3127 catalog_manager.cc:4202] T cd23dbaf55f7426a948169cb684595ce P a180b25a6f8740bfa74c5ecf0ce21d36 reported cstate change: term changed from 0 to 1, leader changed from to a180b25a6f8740bfa74c5ecf0ce21d36 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "a180b25a6f8740bfa74c5ecf0ce21d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a180b25a6f8740bfa74c5ecf0ce21d36" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 39743 } health_report { overall_health: HEALTHY } } } I0521 22:30:44.908082 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver with pid 3161 I0521 22:30:44.930272 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master with pid 3098 2020-05-21T22:30:44Z chronyd exiting [ OK ] ToolTest.TestScanTablePredicates (25969 ms) [ RUN ] ToolTest.TestColumnSetDefault 2020-05-21T22:30:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-05-21T22:30:44Z Disabled control of system clock I0521 22:30:45.001765 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --ipki_ca_key_size=768 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:38089 --webserver_interface=127.0.89.254 --webserver_port=0 --builtin_ntp_servers=127.0.89.212:42113 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:30:45.154609 3499 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:30:45.155146 3499 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:30:45.155634 3499 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:30:45.168058 3499 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=768 W0521 22:30:45.168259 3499 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:30:45.168503 3499 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0521 22:30:45.168727 3499 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0521 22:30:45.187508 3499 master_runner.cc:93] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:42113 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/wal --ipki_ca_key_size=768 --ipki_server_key_size=768 --openssl_security_level_override=0 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:38089 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.89.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:30:45.188380 3499 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:30:45.189884 3499 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:30:45.205003 3499 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:30:45.207976 3499 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:30:45.209301 3499 hybrid_clock.cc:590] HybridClock initialized: now 1590100245209268 us; error 39 us; skew 500 ppm I0521 22:30:45.211122 3499 fs_manager.cc:259] Metadata directory not provided I0521 22:30:45.211275 3499 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:30:45.211625 3499 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:30:45.214821 3499 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/data/instance: uuid: "f96de61ebb214f5ab7aff2dfe9487589" format_stamp: "Formatted at 2020-05-21 22:30:45 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:45.215675 3499 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/wal/instance: uuid: "f96de61ebb214f5ab7aff2dfe9487589" format_stamp: "Formatted at 2020-05-21 22:30:45 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:45.235426 3499 fs_manager.cc:518] Time spent creating directory manager: real 0.019s user 0.017s sys 0.001s I0521 22:30:45.243639 3499 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:30:45.243786 3499 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/wal uuid: "f96de61ebb214f5ab7aff2dfe9487589" format_stamp: "Formatted at 2020-05-21 22:30:45 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:45.244120 3499 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:30:45.999506 3499 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.727s user 0.303s sys 0.423s W0521 22:30:45.999806 3499 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.728s user 0.303s sys 0.423s I0521 22:30:46.048182 3499 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:30:46.117424 3499 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.254:38089 I0521 22:30:46.120959 3499 webserver.cc:381] Webserver started at http://127.0.89.254:36109/ using document root and password file I0521 22:30:46.121903 3499 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/master-0/data/info.pb I0521 22:30:46.126415 3561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:46.131130 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master as pid 3499 I0521 22:30:46.132854 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --rpc_bind_addresses=127.0.89.193:0 --local_ip_for_outbound_sockets=127.0.89.193 --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:38089 --builtin_ntp_servers=127.0.89.212:42113 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} I0521 22:30:46.160538 3561 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: Bootstrap starting. I0521 22:30:46.165397 3561 tablet_bootstrap.cc:596] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: No blocks or log segments found. Creating new log. I0521 22:30:46.167203 3561 log.cc:819] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: Log is configured to *not* fsync() on all Append() calls I0521 22:30:46.171883 3561 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: No bootstrap required, opened a new log I0521 22:30:46.187319 3561 raft_consensus.cc:342] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96de61ebb214f5ab7aff2dfe9487589" member_type: VOTER } I0521 22:30:46.187747 3561 raft_consensus.cc:368] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:46.187944 3561 raft_consensus.cc:722] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96de61ebb214f5ab7aff2dfe9487589, State: Initialized, Role: FOLLOWER I0521 22:30:46.188695 3561 consensus_queue.cc:250] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96de61ebb214f5ab7aff2dfe9487589" member_type: VOTER } I0521 22:30:46.189122 3561 raft_consensus.cc:382] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:46.189384 3561 raft_consensus.cc:478] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:46.189647 3561 raft_consensus.cc:2975] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:46.192968 3561 raft_consensus.cc:500] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96de61ebb214f5ab7aff2dfe9487589" member_type: VOTER } I0521 22:30:46.193529 3561 leader_election.cc:302] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f96de61ebb214f5ab7aff2dfe9487589; no voters: I0521 22:30:46.196303 3561 leader_election.cc:288] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [CANDIDATE]: Term 1 election: Requested vote from peers WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:30:46.327965 3562 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:30:46.328629 3562 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:30:46.329558 3562 flags.cc:403] Enabled unsafe flag: --never_fsync=true I0521 22:30:46.196593 3564 raft_consensus.cc:2721] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [term 1 FOLLOWER]: Leader election won for term 1 W0521 22:30:46.351320 3562 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 I0521 22:30:46.352360 3564 raft_consensus.cc:683] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [term 1 LEADER]: Becoming Leader. State: Replica: f96de61ebb214f5ab7aff2dfe9487589, State: Running, Role: LEADER I0521 22:30:46.352954 3564 consensus_queue.cc:227] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96de61ebb214f5ab7aff2dfe9487589" member_type: VOTER } W0521 22:30:46.353211 3562 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.89.193 I0521 22:30:46.357554 3561 sys_catalog.cc:458] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [sys.catalog]: configured and running, proceeding with master startup. I0521 22:30:46.357825 3565 sys_catalog.cc:357] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f96de61ebb214f5ab7aff2dfe9487589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96de61ebb214f5ab7aff2dfe9487589" member_type: VOTER } } I0521 22:30:46.357978 3566 sys_catalog.cc:357] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f96de61ebb214f5ab7aff2dfe9487589. Latest consensus state: current_term: 1 leader_uuid: "f96de61ebb214f5ab7aff2dfe9487589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f96de61ebb214f5ab7aff2dfe9487589" member_type: VOTER } } I0521 22:30:46.358861 3565 sys_catalog.cc:360] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [sys.catalog]: This master's current role is: LEADER I0521 22:30:46.358866 3566 sys_catalog.cc:360] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589 [sys.catalog]: This master's current role is: LEADER I0521 22:30:46.376401 3562 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:42113 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/wal --ipki_server_key_size=768 --openssl_security_level_override=0 --rpc_bind_addresses=127.0.89.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:38089 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.89.193 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:30:46.377655 3562 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:30:46.379364 3562 file_cache.cc:477] Constructed file cache file cache with capacity 419430 W0521 22:30:46.380635 3570 catalog_manager.cc:1201] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0521 22:30:46.380892 3570 catalog_manager.cc:667] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0521 22:30:46.381238 3573 catalog_manager.cc:1131] Loading table and tablet metadata into memory... I0521 22:30:46.386869 3573 catalog_manager.cc:1142] Initializing Kudu internal certificate authority... I0521 22:30:46.404217 3562 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:30:46.412463 3562 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:30:46.415940 3562 hybrid_clock.cc:590] HybridClock initialized: now 1590100246415916 us; error 32 us; skew 500 ppm I0521 22:30:46.417737 3562 fs_manager.cc:259] Metadata directory not provided I0521 22:30:46.417906 3562 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:30:46.418233 3562 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:30:46.421809 3562 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/data/instance: uuid: "38b7d425b9764072b70c9cff01a53404" format_stamp: "Formatted at 2020-05-21 22:30:46 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:46.422616 3562 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/wal/instance: uuid: "38b7d425b9764072b70c9cff01a53404" format_stamp: "Formatted at 2020-05-21 22:30:46 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:46.426911 3573 catalog_manager.cc:1026] Generated new certificate authority record I0521 22:30:46.429239 3573 catalog_manager.cc:1151] Loading token signing keys... I0521 22:30:46.449193 3573 catalog_manager.cc:4538] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: Generated new TSK 0 I0521 22:30:46.450006 3573 catalog_manager.cc:1161] Initializing in-progress tserver states... I0521 22:30:46.457401 3562 fs_manager.cc:518] Time spent creating directory manager: real 0.034s user 0.035s sys 0.000s I0521 22:30:46.465366 3562 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:30:46.465502 3562 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/wal uuid: "38b7d425b9764072b70c9cff01a53404" format_stamp: "Formatted at 2020-05-21 22:30:46 on dist-test-slave-dist-test-slave-jp1g" I0521 22:30:46.465788 3562 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:30:47.510129 3562 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 1.010s user 0.000s sys 0.002s W0521 22:30:47.510416 3562 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 1.010s user 0.000s sys 0.003s I0521 22:30:47.555631 3562 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:30:47.560432 3562 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:30:47.560586 3562 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:30:47.900368 3562 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.193:37843 I0521 22:30:47.904417 3562 webserver.cc:381] Webserver started at http://127.0.89.193:43271/ using document root and password file I0521 22:30:47.905266 3562 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestColumnSetDefault.1590100104622499-359/minicluster-data/ts-0/data/info.pb I0521 22:30:47.906355 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver as pid 3562 I0521 22:30:47.942728 3694 heartbeater.cc:325] Connected to a master server at 127.0.89.254:38089 I0521 22:30:47.943128 3694 heartbeater.cc:416] Registering TS with master... I0521 22:30:47.944094 3694 heartbeater.cc:465] Master 127.0.89.254:38089 requested a full tablet report, sending... I0521 22:30:47.946563 3528 ts_manager.cc:191] Registered new tserver with Master: 38b7d425b9764072b70c9cff01a53404 (127.0.89.193:37843) I0521 22:30:47.948781 3528 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.89.193:55429 I0521 22:30:47.950520 359 external_mini_cluster.cc:699] 1 TS(s) registered with all masters I0521 22:30:47.953984 359 test_util.cc:232] Using random seed: 275759667 I0521 22:30:47.999853 3528 catalog_manager.cc:1501] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35114: name: "kudu.table.set.default" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "col.int" type: INT64 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "col.string" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "col.bool" type: BOOL is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "col.float" type: FLOAT is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "col.double" type: DOUBLE is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "col.binary" type: BINARY is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "col.unixtime_micros" type: UNIXTIME_MICROS is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "col.decimal" type: DECIMAL128 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 type_attributes { precision: 30 scale: 4 } } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0521 22:30:48.052251 3631 tablet_service.cc:1232] Processing CreateTablet for tablet e99c805460244d78b05218f8a299d4a8 (table=kudu.table.set.default [id=33b2ba06fd284cdc8095df02eebd620f]), partition=RANGE (key) PARTITION UNBOUNDED I0521 22:30:48.053606 3631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e99c805460244d78b05218f8a299d4a8. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:30:48.080880 3711 tablet_bootstrap.cc:440] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Bootstrap starting. I0521 22:30:48.085434 3711 tablet_bootstrap.cc:596] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: No blocks or log segments found. Creating new log. I0521 22:30:48.087285 3711 log.cc:819] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Log is configured to *not* fsync() on all Append() calls I0521 22:30:48.102619 3711 tablet_bootstrap.cc:440] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: No bootstrap required, opened a new log I0521 22:30:48.102962 3711 ts_tablet_manager.cc:1115] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.000s I0521 22:30:48.116333 3711 raft_consensus.cc:342] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b7d425b9764072b70c9cff01a53404" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37843 } } I0521 22:30:48.116832 3711 raft_consensus.cc:368] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:30:48.117023 3711 raft_consensus.cc:722] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b7d425b9764072b70c9cff01a53404, State: Initialized, Role: FOLLOWER I0521 22:30:48.117558 3711 consensus_queue.cc:250] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b7d425b9764072b70c9cff01a53404" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37843 } } I0521 22:30:48.117974 3711 raft_consensus.cc:382] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:30:48.118196 3711 raft_consensus.cc:478] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:30:48.118436 3711 raft_consensus.cc:2975] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:30:48.121609 3711 raft_consensus.cc:500] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b7d425b9764072b70c9cff01a53404" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37843 } } I0521 22:30:48.122236 3711 leader_election.cc:302] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b7d425b9764072b70c9cff01a53404; no voters: I0521 22:30:48.124358 3711 leader_election.cc:288] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:30:48.124675 3713 raft_consensus.cc:2721] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:30:48.305883 3694 heartbeater.cc:457] Master 127.0.89.254:38089 was elected leader, sending a full tablet report... I0521 22:30:48.305941 3711 ts_tablet_manager.cc:1144] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Time spent starting tablet: real 0.203s user 0.100s sys 0.104s I0521 22:30:48.306915 3713 raft_consensus.cc:683] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [term 1 LEADER]: Becoming Leader. State: Replica: 38b7d425b9764072b70c9cff01a53404, State: Running, Role: LEADER I0521 22:30:48.307462 3713 consensus_queue.cc:227] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b7d425b9764072b70c9cff01a53404" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37843 } } I0521 22:30:48.317075 3528 catalog_manager.cc:4202] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404 reported cstate change: term changed from 0 to 1, leader changed from to 38b7d425b9764072b70c9cff01a53404 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "38b7d425b9764072b70c9cff01a53404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b7d425b9764072b70c9cff01a53404" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 37843 } health_report { overall_health: HEALTHY } } } I0521 22:30:49.455678 3528 catalog_manager.cc:2431] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35120: table { table_name: "kudu.table.set.default" } alter_schema_steps { type: ALTER_COLUMN alter_column { delta { name: "col.int" default_value: "\376\377\377\377\377\377\377\377" } } } modify_external_catalogs: true I0521 22:30:49.488515 3743 tablet.cc:1269] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Alter schema from ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 0 to ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 1 I0521 22:30:49.490561 3743 tablet.cc:1208] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: MemRowSet was empty: no flush needed. I0521 22:30:49.507086 3528 catalog_manager.cc:4691] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: kudu.table.set.default [id=33b2ba06fd284cdc8095df02eebd620f] alter complete (version 1) I0521 22:30:56.036315 3528 catalog_manager.cc:2431] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35126: table { table_name: "kudu.table.set.default" } alter_schema_steps { type: ALTER_COLUMN alter_column { delta { name: "col.string" default_value: "string_value" } } } modify_external_catalogs: true I0521 22:30:56.073138 3792 tablet.cc:1269] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Alter schema from ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 1 to ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 2 I0521 22:30:56.075536 3792 tablet.cc:1208] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: MemRowSet was empty: no flush needed. I0521 22:30:56.100895 3528 catalog_manager.cc:4691] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: kudu.table.set.default [id=33b2ba06fd284cdc8095df02eebd620f] alter complete (version 2) I0521 22:31:00.336417 3528 catalog_manager.cc:2431] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35130: table { table_name: "kudu.table.set.default" } alter_schema_steps { type: ALTER_COLUMN alter_column { delta { name: "col.string" default_value: "" } } } modify_external_catalogs: true I0521 22:31:00.372725 3827 tablet.cc:1269] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Alter schema from ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 2 to ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 3 I0521 22:31:00.374945 3827 tablet.cc:1208] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: MemRowSet was empty: no flush needed. I0521 22:31:00.390892 3528 catalog_manager.cc:4691] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: kudu.table.set.default [id=33b2ba06fd284cdc8095df02eebd620f] alter complete (version 3) I0521 22:31:02.492233 3528 catalog_manager.cc:2431] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35132: table { table_name: "kudu.table.set.default" } alter_schema_steps { type: ALTER_COLUMN alter_column { delta { name: "col.string" default_value: "" } } } modify_external_catalogs: true I0521 22:31:02.533402 3848 tablet.cc:1269] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Alter schema from ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 3 to ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 4 I0521 22:31:02.548182 3528 catalog_manager.cc:4691] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: kudu.table.set.default [id=33b2ba06fd284cdc8095df02eebd620f] alter complete (version 4) I0521 22:31:08.886711 3528 catalog_manager.cc:2431] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35138: table { table_name: "kudu.table.set.default" } alter_schema_steps { type: ALTER_COLUMN alter_column { delta { name: "col.bool" default_value: "\001\000\000\000\000\000\000\000" } } } modify_external_catalogs: true I0521 22:31:08.912724 3898 tablet.cc:1269] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Alter schema from ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 4 to ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 5 I0521 22:31:08.914587 3898 tablet.cc:1208] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: MemRowSet was empty: no flush needed. I0521 22:31:08.930510 3528 catalog_manager.cc:4691] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: kudu.table.set.default [id=33b2ba06fd284cdc8095df02eebd620f] alter complete (version 5) I0521 22:31:11.035588 3528 catalog_manager.cc:2431] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35140: table { table_name: "kudu.table.set.default" } alter_schema_steps { type: ALTER_COLUMN alter_column { delta { name: "col.bool" default_value: "\000\000\000\000\000\000\000\000" } } } modify_external_catalogs: true I0521 22:31:11.092664 3919 tablet.cc:1269] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Alter schema from ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 5 to ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 6 I0521 22:31:11.095173 3919 tablet.cc:1208] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: MemRowSet was empty: no flush needed. I0521 22:31:11.110764 3528 catalog_manager.cc:4691] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: kudu.table.set.default [id=33b2ba06fd284cdc8095df02eebd620f] alter complete (version 6) I0521 22:31:15.926219 3528 catalog_manager.cc:2431] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35144: table { table_name: "kudu.table.set.default" } alter_schema_steps { type: ALTER_COLUMN alter_column { delta { name: "col.float" default_value: "\244p\235?" } } } modify_external_catalogs: true I0521 22:31:15.952235 3955 tablet.cc:1269] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Alter schema from ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 6 to ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 7 I0521 22:31:15.955077 3955 tablet.cc:1208] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: MemRowSet was empty: no flush needed. I0521 22:31:15.974189 3527 catalog_manager.cc:4691] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: kudu.table.set.default [id=33b2ba06fd284cdc8095df02eebd620f] alter complete (version 7) I0521 22:31:18.098716 3527 catalog_manager.cc:2431] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35146: table { table_name: "kudu.table.set.default" } alter_schema_steps { type: ALTER_COLUMN alter_column { delta { name: "col.double" default_value: "\215\227n\022\203\300\363\277" } } } modify_external_catalogs: true I0521 22:31:18.133612 3977 tablet.cc:1269] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Alter schema from ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 7 to ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 8 I0521 22:31:18.135464 3977 tablet.cc:1208] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: MemRowSet was empty: no flush needed. I0521 22:31:18.153381 3527 catalog_manager.cc:4691] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: kudu.table.set.default [id=33b2ba06fd284cdc8095df02eebd620f] alter complete (version 8) I0521 22:31:20.495597 3528 catalog_manager.cc:2431] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35148: table { table_name: "kudu.table.set.default" } alter_schema_steps { type: ALTER_COLUMN alter_column { delta { name: "col.binary" default_value: "binary_value" } } } modify_external_catalogs: true I0521 22:31:20.528993 3998 tablet.cc:1269] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Alter schema from ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 8 to ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 9 I0521 22:31:20.530917 3998 tablet.cc:1208] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: MemRowSet was empty: no flush needed. I0521 22:31:20.545529 3528 catalog_manager.cc:4691] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: kudu.table.set.default [id=33b2ba06fd284cdc8095df02eebd620f] alter complete (version 9) I0521 22:31:23.032970 3528 catalog_manager.cc:2431] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35150: table { table_name: "kudu.table.set.default" } alter_schema_steps { type: ALTER_COLUMN alter_column { delta { name: "col.unixtime_micros" default_value: "90\000\000\000\000\000\000" } } } modify_external_catalogs: true I0521 22:31:23.080582 4019 tablet.cc:1269] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: Alter schema from ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 9 to ( 10:key INT32 NOT NULL, 11:col.int INT64 NULLABLE, 12:col.string STRING NULLABLE, 13:col.bool BOOL NULLABLE, 14:col.float FLOAT NULLABLE, 15:col.double DOUBLE NULLABLE, 16:col.binary BINARY NULLABLE, 17:col.unixtime_micros UNIXTIME_MICROS NULLABLE, 18:col.decimal DECIMAL(30, 4) NULLABLE, PRIMARY KEY (key) ) version 10 I0521 22:31:23.082878 4019 tablet.cc:1208] T e99c805460244d78b05218f8a299d4a8 P 38b7d425b9764072b70c9cff01a53404: MemRowSet was empty: no flush needed. I0521 22:31:23.102970 3528 catalog_manager.cc:4691] T 00000000000000000000000000000000 P f96de61ebb214f5ab7aff2dfe9487589: kudu.table.set.default [id=33b2ba06fd284cdc8095df02eebd620f] alter complete (version 10) I0521 22:31:26.267735 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver with pid 3562 I0521 22:31:26.287264 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master with pid 3499 2020-05-21T22:31:26Z chronyd exiting [ OK ] ToolTest.TestColumnSetDefault (41364 ms) [ RUN ] ToolTest.TestHmsList 2020-05-21T22:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-05-21T22:31:26Z Disabled control of system clock Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/derby.log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/tmp/dist-test-taskopyat3/thirdparty/src/hive-3.1.1/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/dist-test-taskopyat3/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Metastore connection URL: jdbc:derby:/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/metadb;create=true Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver Metastore connection User: APP Starting metastore schema initialization to 3.1.0 Initialization script hive-schema-3.1.0.derby.sql Initialization script completed schemaTool completed Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/derby.log 2020-05-21 22:31:33: Starting Hive Metastore Server Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/derby.log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/tmp/dist-test-taskopyat3/thirdparty/src/hive-3.1.1/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/dist-test-taskopyat3/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Starting hive metastore on port 0 I0521 22:31:41.697604 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --ipki_ca_key_size=768 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:42487 --webserver_interface=127.0.89.254 --webserver_port=0 --builtin_ntp_servers=127.0.89.212:46657 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:31:41.850729 4293 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:31:41.851264 4293 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:31:41.851770 4293 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:31:41.864295 4293 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=768 W0521 22:31:41.864472 4293 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:31:41.864675 4293 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0521 22:31:41.864876 4293 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0521 22:31:41.883244 4293 master_runner.cc:93] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:46657 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/wal --ipki_ca_key_size=768 --ipki_server_key_size=768 --openssl_security_level_override=0 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:42487 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.89.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:31:41.884124 4293 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:31:41.885607 4293 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:31:41.900647 4293 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:31:41.904347 4293 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:31:41.905735 4293 hybrid_clock.cc:590] HybridClock initialized: now 1590100301905700 us; error 42 us; skew 500 ppm I0521 22:31:41.907796 4293 fs_manager.cc:259] Metadata directory not provided I0521 22:31:41.907949 4293 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:31:41.908301 4293 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:31:41.911418 4293 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data/instance: uuid: "c3cc85c33a5447b2aa520019fe162966" format_stamp: "Formatted at 2020-05-21 22:31:41 on dist-test-slave-dist-test-slave-jp1g" I0521 22:31:41.912250 4293 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/wal/instance: uuid: "c3cc85c33a5447b2aa520019fe162966" format_stamp: "Formatted at 2020-05-21 22:31:41 on dist-test-slave-dist-test-slave-jp1g" I0521 22:31:41.932162 4293 fs_manager.cc:518] Time spent creating directory manager: real 0.019s user 0.019s sys 0.000s I0521 22:31:41.940260 4293 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:31:41.940400 4293 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/wal uuid: "c3cc85c33a5447b2aa520019fe162966" format_stamp: "Formatted at 2020-05-21 22:31:41 on dist-test-slave-dist-test-slave-jp1g" I0521 22:31:41.940654 4293 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:31:42.722806 4293 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.755s user 0.296s sys 0.440s W0521 22:31:42.723047 4293 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.755s user 0.296s sys 0.440s I0521 22:31:42.746604 4293 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:31:42.820333 4293 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.254:42487 I0521 22:31:42.825227 4293 webserver.cc:381] Webserver started at http://127.0.89.254:43871/ using document root and password file I0521 22:31:42.826658 4293 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data/info.pb I0521 22:31:42.827992 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master as pid 4293 I0521 22:31:42.830545 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --rpc_bind_addresses=127.0.89.193:0 --local_ip_for_outbound_sockets=127.0.89.193 --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:42487 --builtin_ntp_servers=127.0.89.212:46657 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} I0521 22:31:42.832937 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:31:42.861208 4355 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966: Bootstrap starting. I0521 22:31:42.867784 4355 tablet_bootstrap.cc:596] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966: No blocks or log segments found. Creating new log. I0521 22:31:42.869653 4355 log.cc:819] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966: Log is configured to *not* fsync() on all Append() calls I0521 22:31:42.873440 4355 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966: No bootstrap required, opened a new log I0521 22:31:42.888317 4355 raft_consensus.cc:342] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cc85c33a5447b2aa520019fe162966" member_type: VOTER } I0521 22:31:42.888710 4355 raft_consensus.cc:368] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:31:42.888916 4355 raft_consensus.cc:722] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3cc85c33a5447b2aa520019fe162966, State: Initialized, Role: FOLLOWER I0521 22:31:42.889519 4355 consensus_queue.cc:250] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cc85c33a5447b2aa520019fe162966" member_type: VOTER } I0521 22:31:42.889902 4355 raft_consensus.cc:382] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:31:42.890127 4355 raft_consensus.cc:478] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:31:42.890388 4355 raft_consensus.cc:2975] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:31:42.893512 4355 raft_consensus.cc:500] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cc85c33a5447b2aa520019fe162966" member_type: VOTER } I0521 22:31:42.894030 4355 leader_election.cc:302] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3cc85c33a5447b2aa520019fe162966; no voters: I0521 22:31:42.895610 4358 raft_consensus.cc:2721] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 1 FOLLOWER]: Leader election won for term 1 WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:31:43.017310 4356 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:31:43.017983 4356 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:31:43.018806 4356 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:31:43.035740 4356 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:31:43.036779 4356 flags.cc:403] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.89.193 I0521 22:31:43.064343 4356 tablet_server_runner.cc:68] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:46657 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/wal --ipki_server_key_size=768 --openssl_security_level_override=0 --rpc_bind_addresses=127.0.89.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/data/info.pb --webserver_interface=127.0.89.193 --webserver_port=0 --tserver_master_addrs=127.0.89.254:42487 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-tserver --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.89.193 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:31:43.065726 4356 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:31:43.067737 4356 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:31:42.896714 4355 leader_election.cc:288] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:31:43.083209 4358 raft_consensus.cc:683] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 1 LEADER]: Becoming Leader. State: Replica: c3cc85c33a5447b2aa520019fe162966, State: Running, Role: LEADER I0521 22:31:43.084136 4358 consensus_queue.cc:227] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cc85c33a5447b2aa520019fe162966" member_type: VOTER } I0521 22:31:43.090762 4356 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:31:43.094280 4356 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:31:43.095782 4356 hybrid_clock.cc:590] HybridClock initialized: now 1590100303095730 us; error 49 us; skew 500 ppm I0521 22:31:43.098260 4356 fs_manager.cc:259] Metadata directory not provided I0521 22:31:43.098515 4356 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:31:43.099030 4356 server_base.cc:492] This appears to be a new deployment of Kudu; creating new FS layout I0521 22:31:43.099093 4359 sys_catalog.cc:357] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3cc85c33a5447b2aa520019fe162966. Latest consensus state: current_term: 1 leader_uuid: "c3cc85c33a5447b2aa520019fe162966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cc85c33a5447b2aa520019fe162966" member_type: VOTER } } I0521 22:31:43.099925 4359 sys_catalog.cc:360] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [sys.catalog]: This master's current role is: LEADER I0521 22:31:43.099354 4358 sys_catalog.cc:357] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3cc85c33a5447b2aa520019fe162966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cc85c33a5447b2aa520019fe162966" member_type: VOTER } } I0521 22:31:43.100544 4358 sys_catalog.cc:360] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [sys.catalog]: This master's current role is: LEADER I0521 22:31:43.101390 4355 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [sys.catalog]: configured and running, proceeding with master startup. I0521 22:31:43.103873 4356 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/data/instance: uuid: "cf9e08c4271e4d9aa28b1aacbd630908" format_stamp: "Formatted at 2020-05-21 22:31:43 on dist-test-slave-dist-test-slave-jp1g" I0521 22:31:43.105044 4356 fs_manager.cc:617] Generated new instance metadata in path /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/wal/instance: uuid: "cf9e08c4271e4d9aa28b1aacbd630908" format_stamp: "Formatted at 2020-05-21 22:31:43 on dist-test-slave-dist-test-slave-jp1g" W0521 22:31:43.117657 4370 catalog_manager.cc:1201] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0521 22:31:43.117868 4370 catalog_manager.cc:667] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0521 22:31:43.118238 4369 catalog_manager.cc:1131] Loading table and tablet metadata into memory... I0521 22:31:43.123997 4369 catalog_manager.cc:1142] Initializing Kudu internal certificate authority... I0521 22:31:43.137558 4356 fs_manager.cc:518] Time spent creating directory manager: real 0.032s user 0.028s sys 0.004s I0521 22:31:43.151026 4356 fs_manager.cc:433] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0521 22:31:43.151242 4356 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/wal uuid: "cf9e08c4271e4d9aa28b1aacbd630908" format_stamp: "Formatted at 2020-05-21 22:31:43 on dist-test-slave-dist-test-slave-jp1g" I0521 22:31:43.151612 4356 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0521 22:31:43.186192 4369 catalog_manager.cc:1026] Generated new certificate authority record I0521 22:31:43.188891 4369 catalog_manager.cc:1151] Loading token signing keys... I0521 22:31:43.203888 4369 catalog_manager.cc:4538] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966: Generated new TSK 0 I0521 22:31:43.204846 4369 catalog_manager.cc:1161] Initializing in-progress tserver states... W0521 22:31:44.144129 4356 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.952s user 0.001s sys 0.000s W0521 22:31:44.144456 4356 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.952s user 0.002s sys 0.000s I0521 22:31:44.176342 4356 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:31:44.181787 4356 ts_tablet_manager.cc:399] Loaded tablet metadata (0 total tablets, 0 live tablets) I0521 22:31:44.181946 4356 ts_tablet_manager.cc:420] Registered 0 tablets I0521 22:31:44.345577 4356 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.193:43527 I0521 22:31:44.349176 4356 webserver.cc:381] Webserver started at http://127.0.89.193:37035/ using document root and password file I0521 22:31:44.350070 4356 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/ts-0/data/info.pb I0521 22:31:44.359918 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-tserver as pid 4356 I0521 22:31:44.383435 4488 heartbeater.cc:325] Connected to a master server at 127.0.89.254:42487 I0521 22:31:44.383755 4488 heartbeater.cc:416] Registering TS with master... I0521 22:31:44.384799 4488 heartbeater.cc:465] Master 127.0.89.254:42487 requested a full tablet report, sending... I0521 22:31:44.387462 4321 ts_manager.cc:191] Registered new tserver with Master: cf9e08c4271e4d9aa28b1aacbd630908 (127.0.89.193:43527) I0521 22:31:44.390370 359 external_mini_cluster.cc:699] 1 TS(s) registered with all masters I0521 22:31:44.390439 4321 master_service.cc:349] Signed X509 certificate for tserver {username='slave'} at 127.0.89.193:41027 I0521 22:31:44.437877 4321 catalog_manager.cc:1501] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40182: name: "default.simple" schema { columns { name: "foo" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "foo" } } } I0521 22:31:44.501204 4425 tablet_service.cc:1232] Processing CreateTablet for tablet ec39394695764bf38bd187eecb839716 (table=default.simple [id=d78c0e9e374e4c1787ae326a4f8ff676]), partition=RANGE (foo) PARTITION UNBOUNDED I0521 22:31:44.502259 4425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec39394695764bf38bd187eecb839716. 1 dirs total, 0 dirs full, 0 dirs failed I0521 22:31:44.519404 4509 tablet_bootstrap.cc:440] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908: Bootstrap starting. I0521 22:31:44.523663 4509 tablet_bootstrap.cc:596] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908: No blocks or log segments found. Creating new log. I0521 22:31:44.525950 4509 log.cc:819] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908: Log is configured to *not* fsync() on all Append() calls I0521 22:31:44.530289 4509 tablet_bootstrap.cc:440] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908: No bootstrap required, opened a new log I0521 22:31:44.530714 4509 ts_tablet_manager.cc:1115] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s I0521 22:31:44.551609 4509 raft_consensus.cc:342] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9e08c4271e4d9aa28b1aacbd630908" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 43527 } } I0521 22:31:44.552378 4509 raft_consensus.cc:368] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0521 22:31:44.552637 4509 raft_consensus.cc:722] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9e08c4271e4d9aa28b1aacbd630908, State: Initialized, Role: FOLLOWER I0521 22:31:44.553342 4509 consensus_queue.cc:250] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9e08c4271e4d9aa28b1aacbd630908" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 43527 } } I0521 22:31:44.553956 4509 raft_consensus.cc:382] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:31:44.554352 4509 raft_consensus.cc:478] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:31:44.554702 4509 raft_consensus.cc:2975] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [term 0 FOLLOWER]: Advancing to term 1 I0521 22:31:44.559828 4509 raft_consensus.cc:500] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9e08c4271e4d9aa28b1aacbd630908" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 43527 } } I0521 22:31:44.560588 4509 leader_election.cc:302] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf9e08c4271e4d9aa28b1aacbd630908; no voters: I0521 22:31:44.564512 4509 leader_election.cc:288] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [CANDIDATE]: Term 1 election: Requested vote from peers I0521 22:31:44.564888 4511 raft_consensus.cc:2721] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [term 1 FOLLOWER]: Leader election won for term 1 I0521 22:31:44.778702 4509 ts_tablet_manager.cc:1144] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908: Time spent starting tablet: real 0.248s user 0.029s sys 0.008s I0521 22:31:44.779613 4488 heartbeater.cc:457] Master 127.0.89.254:42487 was elected leader, sending a full tablet report... I0521 22:31:44.784168 4511 raft_consensus.cc:683] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [term 1 LEADER]: Becoming Leader. State: Replica: cf9e08c4271e4d9aa28b1aacbd630908, State: Running, Role: LEADER I0521 22:31:44.784973 4511 consensus_queue.cc:227] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9e08c4271e4d9aa28b1aacbd630908" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 43527 } } I0521 22:31:44.793190 4321 catalog_manager.cc:4202] T ec39394695764bf38bd187eecb839716 P cf9e08c4271e4d9aa28b1aacbd630908 reported cstate change: term changed from 0 to 1, leader changed from to cf9e08c4271e4d9aa28b1aacbd630908 (127.0.89.193). New cstate: current_term: 1 leader_uuid: "cf9e08c4271e4d9aa28b1aacbd630908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf9e08c4271e4d9aa28b1aacbd630908" member_type: VOTER last_known_addr { host: "127.0.89.193" port: 43527 } health_report { overall_health: HEALTHY } } } I0521 22:31:47.846518 359 external_mini_cluster.cc:1234] Killing /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master with pid 4293 I0521 22:31:47.866914 359 external_mini_cluster.cc:1016] Running /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --never_fsync --ipki_server_key_size=768 --openssl_security_level_override=0 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --logtostderr --logbuflevel=-1 --ipki_ca_key_size=768 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:42487 --webserver_interface=127.0.89.254 --webserver_port=43871 --builtin_ntp_servers=127.0.89.212:46657 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --rpc_reuseport=true --hive_metastore_uris=thrift://127.0.0.1:45491 --hive_metastore_sasl_enabled=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0521 22:31:48.025044 4530 flags.cc:403] Enabled unsafe flag: --openssl_security_level_override=0 W0521 22:31:48.025553 4530 flags.cc:403] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0521 22:31:48.026041 4530 flags.cc:403] Enabled unsafe flag: --never_fsync=true W0521 22:31:48.038631 4530 flags.cc:403] Enabled experimental flag: --ipki_ca_key_size=768 W0521 22:31:48.038813 4530 flags.cc:403] Enabled experimental flag: --ipki_server_key_size=768 W0521 22:31:48.039024 4530 flags.cc:403] Enabled experimental flag: --tsk_num_rsa_bits=512 W0521 22:31:48.039217 4530 flags.cc:403] Enabled experimental flag: --rpc_reuseport=true I0521 22:31:48.058411 4530 master_runner.cc:93] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.0.89.212:46657 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/wal --hive_metastore_uris=thrift://127.0.0.1:45491 --ipki_ca_key_size=768 --ipki_server_key_size=768 --openssl_security_level_override=0 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.89.254:42487 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.89.254 --webserver_port=43871 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --log_filename=kudu-master --enable_minidumps=false --log_dir=/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.13.0-SNAPSHOT revision 1c6e9a70e819ae2aa7f189c121df80342e1174d1 build type FASTDEBUG built by None at 21 May 2020 22:15:25 UTC on jenkins-slave-snjf build id 21722 TSAN enabled I0521 22:31:48.059264 4530 env_posix.cc:1747] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0521 22:31:48.060806 4530 file_cache.cc:477] Constructed file cache file cache with capacity 419430 I0521 22:31:48.075999 4530 hybrid_clock.cc:243] auto-selected time source: builtin I0521 22:31:48.079511 4530 hybrid_clock.cc:572] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0521 22:31:48.080824 4530 hybrid_clock.cc:590] HybridClock initialized: now 1590100308080796 us; error 36 us; skew 500 ppm I0521 22:31:48.082504 4530 fs_manager.cc:259] Metadata directory not provided I0521 22:31:48.082676 4530 fs_manager.cc:265] Using write-ahead log directory (fs_wal_dir) as metadata directory I0521 22:31:48.098178 4530 fs_manager.cc:410] Time spent opening directory manager: real 0.013s user 0.013s sys 0.000s I0521 22:31:48.105499 4530 fs_manager.cc:433] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0521 22:31:48.105657 4530 fs_manager.cc:451] Opened local filesystem: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data,/tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/wal uuid: "c3cc85c33a5447b2aa520019fe162966" format_stamp: "Formatted at 2020-05-21 22:31:41 on dist-test-slave-dist-test-slave-jp1g" I0521 22:31:48.106954 4530 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0521 22:31:48.825923 4488 heartbeater.cc:581] Failed to heartbeat to 127.0.89.254:42487 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.89.254:42487: connect: Connection refused (error 111) W0521 22:31:49.046895 4530 thread.cc:616] rpc reactor (reactor) Time spent creating pthread: real 0.912s user 0.368s sys 0.544s W0521 22:31:49.047197 4530 thread.cc:582] rpc reactor (reactor) Time spent starting thread: real 0.913s user 0.368s sys 0.544s I0521 22:31:49.064340 4530 env_posix.cc:1747] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0521 22:31:49.141993 4530 rpc_server.cc:221] RPC server started. Bound to: 127.0.89.254:42487 I0521 22:31:49.147094 4530 webserver.cc:381] Webserver started at http://127.0.89.254:43871/ using document root and password file I0521 22:31:49.148650 4530 server_base.cc:675] Dumped server information to /tmp/dist-test-taskopyat3/test-tmp/kudu-tool-test.2.ToolTest.TestHmsList.1590100104622499-359/minicluster-data/master-0/data/info.pb I0521 22:31:49.151304 359 external_mini_cluster.cc:1078] Started /tmp/dist-test-taskopyat3/build/tsan/bin/kudu-master as pid 4530 I0521 22:31:49.167654 4593 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966: Bootstrap starting. I0521 22:31:49.214175 4593 log.cc:819] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966: Log is configured to *not* fsync() on all Append() calls I0521 22:31:49.235616 4593 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0521 22:31:49.236541 4593 tablet_bootstrap.cc:440] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966: Bootstrap complete. I0521 22:31:49.252290 4593 raft_consensus.cc:342] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cc85c33a5447b2aa520019fe162966" member_type: VOTER } I0521 22:31:49.254276 4593 raft_consensus.cc:722] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3cc85c33a5447b2aa520019fe162966, State: Initialized, Role: FOLLOWER I0521 22:31:49.255015 4593 consensus_queue.cc:250] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cc85c33a5447b2aa520019fe162966" member_type: VOTER } I0521 22:31:49.255442 4593 raft_consensus.cc:382] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0521 22:31:49.255760 4593 raft_consensus.cc:478] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0521 22:31:49.256117 4593 raft_consensus.cc:2975] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 1 FOLLOWER]: Advancing to term 2 I0521 22:31:49.263492 4593 raft_consensus.cc:500] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cc85c33a5447b2aa520019fe162966" member_type: VOTER } I0521 22:31:49.264251 4593 leader_election.cc:302] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3cc85c33a5447b2aa520019fe162966; no voters: I0521 22:31:49.266008 4597 raft_consensus.cc:2721] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 2 FOLLOWER]: Leader election won for term 2 I0521 22:31:49.266993 4593 leader_election.cc:288] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [CANDIDATE]: Term 2 election: Requested vote from peers I0521 22:31:49.402870 4597 raft_consensus.cc:683] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [term 2 LEADER]: Becoming Leader. State: Replica: c3cc85c33a5447b2aa520019fe162966, State: Running, Role: LEADER I0521 22:31:49.403853 4597 consensus_queue.cc:227] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cc85c33a5447b2aa520019fe162966" member_type: VOTER } I0521 22:31:49.409034 4598 sys_catalog.cc:357] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3cc85c33a5447b2aa520019fe162966. Latest consensus state: current_term: 2 leader_uuid: "c3cc85c33a5447b2aa520019fe162966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cc85c33a5447b2aa520019fe162966" member_type: VOTER } } I0521 22:31:49.409512 4598 sys_catalog.cc:360] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [sys.catalog]: This master's current role is: LEADER I0521 22:31:49.410904 4593 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [sys.catalog]: configured and running, proceeding with master startup. I0521 22:31:49.410706 4599 sys_catalog.cc:357] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c3cc85c33a5447b2aa520019fe162966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3cc85c33a5447b2aa520019fe162966" member_type: VOTER } } I0521 22:31:49.411566 4599 sys_catalog.cc:360] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966 [sys.catalog]: This master's current role is: LEADER I0521 22:31:49.413431 4601 catalog_manager.cc:1131] Loading table and tablet metadata into memory... I0521 22:31:49.420882 4601 catalog_manager.cc:427] Loaded metadata for table default.simple [id=d78c0e9e374e4c1787ae326a4f8ff676] I0521 22:31:49.425580 4601 catalog_manager.cc:481] Loaded metadata for tablet ec39394695764bf38bd187eecb839716 (table default.simple [id=d78c0e9e374e4c1787ae326a4f8ff676]) I0521 22:31:49.426401 4601 catalog_manager.cc:1142] Initializing Kudu internal certificate authority... I0521 22:31:49.432533 4601 catalog_manager.cc:1151] Loading token signing keys... I0521 22:31:49.435567 4601 catalog_manager.cc:4549] T 00000000000000000000000000000000 P c3cc85c33a5447b2aa520019fe162966: Loaded TSK: 0 I0521 22:31:49.436857 4601 catalog_manager.cc:1161] Initializing in-progress tserver states... I0521 22:31:49.446161 4606 hms_notification_log_listener.cc:228] Skipping Hive Metastore notification log poll: Service unavailable: Catalog manager is not initialized. State: Starting I0521 22:31:49.839709 4488 heartbeater.cc:325] Connected to a master server at 127.0.89.254:42487 I0521 22:31:49.845547 4559 master_service.cc:295] Got heartbeat from unknown tserver (permanent_uuid: "cf9e08c4271e4d9aa28b1aacbd630908" instance_seqno: 1590100304311876) as {username='slave'} at 127.0.89.193:33867; Asking this server to re-register. I0521 22:31:49.846786 4488 heartbeater.cc:416] Registering TS with master... I0521 22:31:49.847297 4488 heartbeater.cc:465] Master 127.0.89.254:42487 requested a full tablet report, sending... I0521 22:31:49.849771 4559 ts_manager.cc:191] Registered new tserver with Master: cf9e08c4271e4d9aa28b1aacbd630908 (127.0.89.193:43527) I0521 22:31:49.852535 359 external_mini_cluster.cc:699] 1 TS(s) registered with all masters W0521 22:32:23.142868 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b060 after lost signal to thread 4531 W0521 22:32:23.143333 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b780 after lost signal to thread 4591 W0521 22:32:28.996440 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b740 after lost signal to thread 4531 W0521 22:32:28.996966 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b520 after lost signal to thread 4591 W0521 22:33:05.743249 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002aae0 after lost signal to thread 4360 W0521 22:33:05.743983 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002af00 after lost signal to thread 4486 I0521 22:33:49.594769 4549 maintenance_manager.cc:326] P c3cc85c33a5447b2aa520019fe162966: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033386 I0521 22:33:49.637208 4548 maintenance_manager.cc:525] P c3cc85c33a5447b2aa520019fe162966: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.042s user 0.032s sys 0.008s Metrics: {"bytes_written":6485,"cfile_init":1,"dirs.queue_time_us":675,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":997,"drs_written":1,"lbm_read_time_us":231,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1980,"lbm_writes_lt_1ms":27,"rows_written":5,"thread_start_us":953,"threads_started":2,"wal-append.queue_time_us":819} I0521 22:33:49.639096 4549 maintenance_manager.cc:326] P c3cc85c33a5447b2aa520019fe162966: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 396 bytes on disk I0521 22:33:49.640486 4548 maintenance_manager.cc:525] P c3cc85c33a5447b2aa520019fe162966: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":269,"lbm_reads_lt_1ms":4} W0521 22:34:17.794472 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002ade0 after lost signal to thread 4360 W0521 22:34:17.795437 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002a7e0 after lost signal to thread 4486 W0521 22:34:20.286921 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b2e0 after lost signal to thread 4531 W0521 22:34:20.287376 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b140 after lost signal to thread 4591 W0521 22:35:27.726336 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002af40 after lost signal to thread 4360 W0521 22:35:27.727084 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002a980 after lost signal to thread 4486 W0521 22:36:12.250830 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b9c0 after lost signal to thread 4531 W0521 22:36:12.251247 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b220 after lost signal to thread 4591 W0521 22:36:34.702325 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b800 after lost signal to thread 4531 W0521 22:36:34.702808 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b700 after lost signal to thread 4591 W0521 22:37:03.237411 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002ab20 after lost signal to thread 4360 W0521 22:37:03.238131 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002a8e0 after lost signal to thread 4486 W0521 22:37:27.446748 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b320 after lost signal to thread 4531 W0521 22:37:27.447190 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080008a460 after lost signal to thread 4591 W0521 22:38:26.852612 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b7c0 after lost signal to thread 4531 W0521 22:38:26.853031 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b8c0 after lost signal to thread 4591 W0521 22:38:28.039356 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002a9a0 after lost signal to thread 4360 W0521 22:38:28.040096 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002ad80 after lost signal to thread 4486 W0521 22:38:45.889757 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002aa60 after lost signal to thread 4360 W0521 22:38:45.890512 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002a0a0 after lost signal to thread 4486 W0521 22:39:57.414119 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002ae60 after lost signal to thread 4360 W0521 22:39:57.414861 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002a140 after lost signal to thread 4486 W0521 22:40:04.007081 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b240 after lost signal to thread 4531 W0521 22:40:04.007515 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b8a0 after lost signal to thread 4591 W0521 22:40:41.883837 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002a880 after lost signal to thread 4360 W0521 22:40:41.884637 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002ace0 after lost signal to thread 4486 W0521 22:41:06.952173 4380 debug-util.cc:402] Leaking SignalData structure 0x7b0800058020 after lost signal to thread 4360 W0521 22:41:06.952941 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002a6e0 after lost signal to thread 4486 W0521 22:41:36.942020 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b560 after lost signal to thread 4531 W0521 22:41:36.942495 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b4a0 after lost signal to thread 4591 W0521 22:42:02.273381 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002bb20 after lost signal to thread 4531 W0521 22:42:02.273813 4545 debug-util.cc:402] Leaking SignalData structure 0x7b080002b0e0 after lost signal to thread 4591 W0521 22:42:37.898852 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002acc0 after lost signal to thread 4360 W0521 22:42:37.899608 4380 debug-util.cc:402] Leaking SignalData structure 0x7b080002aca0 after lost signal to thread 4486 ************************ BEGIN STACKS ************************** [New LWP 4501] [New LWP 4500] [New LWP 4499] [New LWP 4497] [New LWP 4494] [New LWP 4035] [New LWP 389] [New LWP 363] [New LWP 362] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 81 ../sysdeps/unix/syscall-template.S: No such file or directory. Id Target Id Frame 10 Thread 0x7ff88f8ff700 (LWP 362) "kudu-tool-test" 0x00007ff892b06d4d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 9 Thread 0x7ff88f0fe700 (LWP 363) "kudu-tool-test" 0x00007ff89d09ded9 in __libc_waitpid (pid=4917, stat_loc=0x7ff88f0bdde4, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:40 8 Thread 0x7ff88b88f700 (LWP 389) "kernel-watcher-" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 7 Thread 0x7ff8458f7700 (LWP 4035) "rpc reactor-403" 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 6 Thread 0x7ff885cfc700 (LWP 4494) "Hive Metastore " pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 5 Thread 0x7ff88b08e700 (LWP 4497) "rpc reactor-449" 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 4 Thread 0x7ff888ec2700 (LWP 4499) "rpc reactor-449" 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 3 Thread 0x7ff8886c1700 (LWP 4500) "rpc reactor-450" 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 2 Thread 0x7ff887ec0700 (LWP 4501) "rpc reactor-450" 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 * 1 Thread 0x7ff8a61a2400 (LWP 359) "kudu-tool-test" 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 Thread 10 (Thread 0x7ff88f8ff700 (LWP 362)): #0 0x00007ff892b06d4d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007ff892b37fd4 in usleep (useconds=) at ../sysdeps/unix/sysv/linux/usleep.c:32 #2 0x00000000004dc127 in __interceptor_usleep (usec=100000) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:354 #3 0x0000000000537fe8 in __tsan::BackgroundThread (arg=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_rtl.cc:180 #4 0x00007ff89d096184 in start_thread (arg=0x7ff88f8ff700) at pthread_create.c:312 #5 0x00007ff892b3fffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 9 (Thread 0x7ff88f0fe700 (LWP 363)): #0 0x00007ff89d09ded9 in __libc_waitpid (pid=4917, stat_loc=0x7ff88f0bdde4, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:40 #1 0x00000000004e9c7b in __interceptor_waitpid (pid=4917, status=0x7ff88f0bdde4, options=0) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:2408 #2 0x00007ff896d56065 in kudu::Subprocess::DoWait (this=0x7ff88f0bde68, wait_status=0x0, mode=kudu::Subprocess::BLOCKING) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/subprocess.cc:815 #3 0x00007ff896d55f8c in kudu::Subprocess::Wait (this=0x7ff88f0bdde4, wait_status=0x0) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/subprocess.cc:546 #4 0x00007ff896c95c8d in kudu::PstackWatcher::RunStackDump (argv=...) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/pstack_watcher.cc:234 #5 0x00007ff896c95953 in kudu::PstackWatcher::RunGdbStackDump (pid=359, flags=0) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/pstack_watcher.cc:211 #6 0x00007ff896c95418 in kudu::PstackWatcher::DumpPidStacks (pid=359, flags=0) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/pstack_watcher.cc:170 #7 0x00007ff896c94a78 in kudu::PstackWatcher::DumpStacks (flags=0) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/pstack_watcher.cc:162 #8 0x00007ff8a45b4cbe in kudu::CreateAndStartTimeoutThread()::$_0::operator()() const (this=) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/test_main.cc:60 #9 0x00007ff8a45b4c54 in std::__1::__invoke (__f=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/type_traits:3530 #10 0x00007ff8a45b4bc4 in std::__1::__thread_execute >, kudu::CreateAndStartTimeoutThread()::$_0>(std::__1::tuple >, kudu::CreateAndStartTimeoutThread()::$_0>&, std::__1::__tuple_indices<>) (__t=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/thread:277 #11 0x00007ff8a45b495b in std::__1::__thread_proxy >, kudu::CreateAndStartTimeoutThread()::$_0> >(void*) (__vp=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/thread:287 #12 0x00000000004d0580 in __tsan_thread_start_func (arg=0x7ffcdcd2d820) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:957 #13 0x00007ff89d096184 in start_thread (arg=0x7ff88f0fe700) at pthread_create.c:312 #14 0x00007ff892b3fffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 8 (Thread 0x7ff88b88f700 (LWP 389)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000054fa88 in __tsan::call_pthread_cancel_with_cleanup (fn=fn@entry=0x7ff89d09a680 , c=c@entry=0x7b4400000488, m=m@entry=0x7b4400000450, abstime=abstime@entry=0x7ff88b84ed28, cleanup=cleanup@entry=0x4dcdb0 , arg=arg@entry=0x7ff88b84ec50) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cc:450 #2 0x00000000004db06d in cond_wait (thr=thr@entry=0x7ff88b84fb80, pc=5092321, si=si@entry=0x7ff88b84ecc0, fn=0x7ff89d09a680 , c=c@entry=0x7b4400000488, m=m@entry=0x7b4400000450, t=t@entry=0x7ff88b84ed28) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1165 #3 0x00000000004db598 in __interceptor_pthread_cond_timedwait (c=, m=0x7b4400000450, abstime=0x7ff88b84ed28) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1184 #4 0x00007ff896c05d28 in kudu::ConditionVariable::WaitUntil (this=0x7b4400000488, until=...) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/condition_variable.cc:87 #5 0x00007ff8a3a067a9 in kudu::CountDownLatch::WaitUntil (this=, when=...) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/countdown_latch.h:87 #6 0x00007ff8a3a037cf in kudu::CountDownLatch::WaitFor (this=0x7b4400000450, delta=...) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/countdown_latch.h:97 #7 0x00007ff896ca8bb4 in kudu::KernelStackWatchdog::RunThread (this=0x7b44000003c0) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/kernel_stack_watchdog.cc:134 #8 0x00007ff896caa2f2 in kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0::operator()() const (this=0x7b4400000e18) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/kernel_stack_watchdog.cc:71 #9 0x00007ff896caa282 in std::__1::__invoke (__f=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/type_traits:3530 #10 0x00007ff896caa212 in std::__1::__invoke_void_return_wrapper::__call(kudu::KernelStackWatchdog::KernelStackWatchdog()::$_0&) (__args=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/__functional_base:348 #11 0x00007ff896caa1d2 in std::__1::__function::__alloc_func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1533 #12 0x00007ff896ca926e in std::__1::__function::__func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1707 #13 0x00007ff8a3f80a95 in std::__1::__function::__value_func::operator()() const (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1860 #14 0x00007ff8a3f7600a in std::__1::function::operator()() const (this=0x7b440000048c) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:2419 #15 0x00007ff896d6653b in kudu::Thread::SuperviseThread (arg=) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.cc:674 #16 0x00000000004d0580 in __tsan_thread_start_func (arg=0x7ffcdcd2c040) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:957 #17 0x00007ff89d096184 in start_thread (arg=0x7ff88b88f700) at pthread_create.c:312 #18 0x00007ff892b3fffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 7 (Thread 0x7ff8458f7700 (LWP 4035)): #0 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000004db89e in __interceptor_epoll_wait (epfd=8, ev=0x7b580000cf00, cnt=64, timeout=99) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1829 #2 0x00007ff89571275d in epoll_poll (loop=0x7b640001ef00, timeout=0.099717139158201462) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev_epoll.c:153 #3 0x00007ff8957068c2 in ev_run (loop=0x7b640001ef00, flags=0) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev.c:3521 #4 0x00007ff898d8f539 in ev::loop_ref::run (this=0x7b4c00004818, flags=52992) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/ev++.h:211 #5 0x00007ff898d86e4c in kudu::rpc::ReactorThread::RunThread (this=0x7b4c00004810) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:503 #6 0x00007ff898d89eb2 in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=0x7b4400012b18) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:194 #7 0x00007ff898d89e42 in std::__1::__invoke (__f=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/type_traits:3530 #8 0x00007ff898d89dd2 in std::__1::__invoke_void_return_wrapper::__call(kudu::rpc::ReactorThread::Init()::$_0&) (__args=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/__functional_base:348 #9 0x00007ff898d89d92 in std::__1::__function::__alloc_func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1533 #10 0x00007ff898d88e2e in std::__1::__function::__func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1707 #11 0x00007ff8a3f80a95 in std::__1::__function::__value_func::operator()() const (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1860 #12 0x00007ff8a3f7600a in std::__1::function::operator()() const (this=0x8) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:2419 #13 0x00007ff896d6653b in kudu::Thread::SuperviseThread (arg=) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.cc:674 #14 0x00000000004d0580 in __tsan_thread_start_func (arg=0x7ffcdcd2c200) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:957 #15 0x00007ff89d096184 in start_thread (arg=0x7ff8458f7700) at pthread_create.c:312 #16 0x00007ff892b3fffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 6 (Thread 0x7ff885cfc700 (LWP 4494)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000054fa88 in __tsan::call_pthread_cancel_with_cleanup (fn=fn@entry=0x7ff89d09a340 , c=c@entry=0x7ff885cbbe60, m=m@entry=0x7b500009fc38, abstime=abstime@entry=0x0, cleanup=cleanup@entry=0x4dcdb0 , arg=arg@entry=0x7ff885cbbd40) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_platform_linux.cc:450 #2 0x00000000004db06d in cond_wait (thr=thr@entry=0x7ff885cbcb80, pc=5091662, si=si@entry=0x7ff885cbbdb0, fn=0x7ff89d09a340 , c=c@entry=0x7ff885cbbe60, m=m@entry=0x7b500009fc38, t=t@entry=0x0) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1165 #3 0x00000000004db2e4 in __interceptor_pthread_cond_wait (c=, m=0x7b500009fc38) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1177 #4 0x00007ff896c05c08 in kudu::ConditionVariable::Wait (this=0x7ff885cbbe60) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/condition_variable.cc:55 #5 0x00007ff896d7ee47 in kudu::ThreadPool::DispatchThread (this=0x7b500009fc00) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/threadpool.cc:582 #6 0x00007ff896d80fc2 in kudu::ThreadPool::CreateThread()::$_1::operator()() const (this=0x7b4400014cd8) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/threadpool.cc:700 #7 0x00007ff896d80f52 in std::__1::__invoke (__f=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/type_traits:3530 #8 0x00007ff896d80ee2 in std::__1::__invoke_void_return_wrapper::__call(kudu::ThreadPool::CreateThread()::$_1&) (__args=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/__functional_base:348 #9 0x00007ff896d80ea2 in std::__1::__function::__alloc_func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1533 #10 0x00007ff896d7ff3e in std::__1::__function::__func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1707 #11 0x00007ff8a3f80a95 in std::__1::__function::__value_func::operator()() const (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1860 #12 0x00007ff8a3f7600a in std::__1::function::operator()() const (this=0x7ff885cbbe64) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:2419 #13 0x00007ff896d6653b in kudu::Thread::SuperviseThread (arg=) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.cc:674 #14 0x00000000004d0580 in __tsan_thread_start_func (arg=0x7ffcdcd2c610) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:957 #15 0x00007ff89d096184 in start_thread (arg=0x7ff885cfc700) at pthread_create.c:312 #16 0x00007ff892b3fffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 5 (Thread 0x7ff88b08e700 (LWP 4497)): #0 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000004db89e in __interceptor_epoll_wait (epfd=18, ev=0x7b5800027000, cnt=64, timeout=99) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1829 #2 0x00007ff89571275d in epoll_poll (loop=0x7b640001f400, timeout=0.099012609154101483) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev_epoll.c:153 #3 0x00007ff8957068c2 in ev_run (loop=0x7b640001f400, flags=0) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev.c:3521 #4 0x00007ff898d8f539 in ev::loop_ref::run (this=0x7b4c00008558, flags=159744) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/ev++.h:211 #5 0x00007ff898d86e4c in kudu::rpc::ReactorThread::RunThread (this=0x7b4c00008550) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:503 #6 0x00007ff898d89eb2 in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=0x7b4400014e18) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:194 #7 0x00007ff898d89e42 in std::__1::__invoke (__f=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/type_traits:3530 #8 0x00007ff898d89dd2 in std::__1::__invoke_void_return_wrapper::__call(kudu::rpc::ReactorThread::Init()::$_0&) (__args=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/__functional_base:348 #9 0x00007ff898d89d92 in std::__1::__function::__alloc_func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1533 #10 0x00007ff898d88e2e in std::__1::__function::__func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1707 #11 0x00007ff8a3f80a95 in std::__1::__function::__value_func::operator()() const (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1860 #12 0x00007ff8a3f7600a in std::__1::function::operator()() const (this=0x12) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:2419 #13 0x00007ff896d6653b in kudu::Thread::SuperviseThread (arg=) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.cc:674 #14 0x00000000004d0580 in __tsan_thread_start_func (arg=0x7ffcdcd2c520) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:957 #15 0x00007ff89d096184 in start_thread (arg=0x7ff88b08e700) at pthread_create.c:312 #16 0x00007ff892b3fffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 4 (Thread 0x7ff888ec2700 (LWP 4499)): #0 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000004db89e in __interceptor_epoll_wait (epfd=19, ev=0x7b5800027600, cnt=64, timeout=99) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1829 #2 0x00007ff89571275d in epoll_poll (loop=0x7b640001f900, timeout=0.099111404154086813) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev_epoll.c:153 #3 0x00007ff8957068c2 in ev_run (loop=0x7b640001f900, flags=0) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev.c:3521 #4 0x00007ff898d8f539 in ev::loop_ref::run (this=0x7b4c00008398, flags=161280) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/ev++.h:211 #5 0x00007ff898d86e4c in kudu::rpc::ReactorThread::RunThread (this=0x7b4c00008390) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:503 #6 0x00007ff898d89eb2 in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=0x7b4400015318) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:194 #7 0x00007ff898d89e42 in std::__1::__invoke (__f=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/type_traits:3530 #8 0x00007ff898d89dd2 in std::__1::__invoke_void_return_wrapper::__call(kudu::rpc::ReactorThread::Init()::$_0&) (__args=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/__functional_base:348 #9 0x00007ff898d89d92 in std::__1::__function::__alloc_func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1533 #10 0x00007ff898d88e2e in std::__1::__function::__func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1707 #11 0x00007ff8a3f80a95 in std::__1::__function::__value_func::operator()() const (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1860 #12 0x00007ff8a3f7600a in std::__1::function::operator()() const (this=0x13) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:2419 #13 0x00007ff896d6653b in kudu::Thread::SuperviseThread (arg=) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.cc:674 #14 0x00000000004d0580 in __tsan_thread_start_func (arg=0x7ffcdcd2c520) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:957 #15 0x00007ff89d096184 in start_thread (arg=0x7ff888ec2700) at pthread_create.c:312 #16 0x00007ff892b3fffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 3 (Thread 0x7ff8886c1700 (LWP 4500)): #0 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000004db89e in __interceptor_epoll_wait (epfd=20, ev=0x7b5800027300, cnt=64, timeout=99) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1829 #2 0x00007ff89571275d in epoll_poll (loop=0x7b64000d7000, timeout=0.099264657154094493) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev_epoll.c:153 #3 0x00007ff8957068c2 in ev_run (loop=0x7b64000d7000, flags=0) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev.c:3521 #4 0x00007ff898d8f539 in ev::loop_ref::run (this=0x7b4c000081d8, flags=160512) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/ev++.h:211 #5 0x00007ff898d86e4c in kudu::rpc::ReactorThread::RunThread (this=0x7b4c000081d0) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:503 #6 0x00007ff898d89eb2 in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=0x7b4400012ed8) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:194 #7 0x00007ff898d89e42 in std::__1::__invoke (__f=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/type_traits:3530 #8 0x00007ff898d89dd2 in std::__1::__invoke_void_return_wrapper::__call(kudu::rpc::ReactorThread::Init()::$_0&) (__args=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/__functional_base:348 #9 0x00007ff898d89d92 in std::__1::__function::__alloc_func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1533 #10 0x00007ff898d88e2e in std::__1::__function::__func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1707 #11 0x00007ff8a3f80a95 in std::__1::__function::__value_func::operator()() const (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1860 #12 0x00007ff8a3f7600a in std::__1::function::operator()() const (this=0x14) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:2419 #13 0x00007ff896d6653b in kudu::Thread::SuperviseThread (arg=) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.cc:674 #14 0x00000000004d0580 in __tsan_thread_start_func (arg=0x7ffcdcd2c520) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:957 #15 0x00007ff89d096184 in start_thread (arg=0x7ff8886c1700) at pthread_create.c:312 #16 0x00007ff892b3fffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 2 (Thread 0x7ff887ec0700 (LWP 4501)): #0 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000004db89e in __interceptor_epoll_wait (epfd=21, ev=0x7b5800026d00, cnt=64, timeout=99) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1829 #2 0x00007ff89571275d in epoll_poll (loop=0x7b64000d7500, timeout=0.09933258415412638) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev_epoll.c:153 #3 0x00007ff8957068c2 in ev_run (loop=0x7b64000d7500, flags=0) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev.c:3521 #4 0x00007ff898d8f539 in ev::loop_ref::run (this=0x7b4c00008018, flags=158976) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/ev++.h:211 #5 0x00007ff898d86e4c in kudu::rpc::ReactorThread::RunThread (this=0x7b4c00008010) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:503 #6 0x00007ff898d89eb2 in kudu::rpc::ReactorThread::Init()::$_0::operator()() const (this=0x7b4400014a58) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/rpc/reactor.cc:194 #7 0x00007ff898d89e42 in std::__1::__invoke (__f=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/type_traits:3530 #8 0x00007ff898d89dd2 in std::__1::__invoke_void_return_wrapper::__call(kudu::rpc::ReactorThread::Init()::$_0&) (__args=...) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/__functional_base:348 #9 0x00007ff898d89d92 in std::__1::__function::__alloc_func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1533 #10 0x00007ff898d88e2e in std::__1::__function::__func, void ()>::operator()() (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1707 #11 0x00007ff8a3f80a95 in std::__1::__function::__value_func::operator()() const (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:1860 #12 0x00007ff8a3f7600a in std::__1::function::operator()() const (this=0x15) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/c++/v1/functional:2419 #13 0x00007ff896d6653b in kudu::Thread::SuperviseThread (arg=) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/thread.cc:674 #14 0x00000000004d0580 in __tsan_thread_start_func (arg=0x7ffcdcd2c520) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:957 #15 0x00007ff89d096184 in start_thread (arg=0x7ff887ec0700) at pthread_create.c:312 #16 0x00007ff892b3fffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7ff8a61a2400 (LWP 359)): #0 0x00007ff892b40693 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000004db89e in __interceptor_epoll_wait (epfd=31, ev=0x7b580000c900, cnt=64, timeout=59743) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1829 #2 0x00007ff89571275d in epoll_poll (loop=0x7b64000d7a00, timeout=59.743000000000002) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev_epoll.c:153 #3 0x00007ff8957068c2 in ev_run (loop=0x7b64000d7a00, flags=0) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/libev-4.20/ev.c:3521 #4 0x00007ff898d8f539 in ev::loop_ref::run (this=0x7ffcdcd2ca40, flags=51456) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/ev++.h:211 #5 0x00007ff896d57b1d in kudu::(anonymous namespace)::ReadFdsFully (progname=..., fds=..., out=0x7ffcdcd2cb70) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/subprocess.cc:242 #6 0x00007ff896d5754a in kudu::Subprocess::Call (argv=..., stdin_in=..., stdout_out=, stderr_out=0x7ffcdcd2cd78, env_vars=...) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/subprocess.cc:771 #7 0x00007ff8a5781532 in kudu::tools::RunKuduTool (args=..., out=0x7ffcdcd2cd60, err=0x7ffcdcd2cd78, in=...) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/tools/tool_test_util.cc:77 #8 0x00000000005f0905 in kudu::tools::ToolTest::RunTool (this=, arg_str=..., stdout=0x7ffcdcd2d580, stderr=0x7ffcdcd2cdf8, stdout_lines=0x0, stderr_lines=0x0, in=...) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/tools/kudu-tool-test.cc:246 #9 0x00000000005f52ea in kudu::tools::ToolTest::RunActionStdoutString (this=0x7b2000001080, arg_str=..., stdout=0x7ffcdcd2d580) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/tools/kudu-tool-test.cc:283 #10 0x00000000005cb481 in kudu::tools::ToolTest_TestHmsList_Test::TestBody (this=0x7b2000001080) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/tools/kudu-tool-test.cc:4588 #11 0x00007ff8a3d32fa0 in HandleSehExceptionsInMethodIfSupported (object=0x7b2000001080, method=, location=0x7ff8a3d50c8a "the test body") at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402 #12 testing::internal::HandleExceptionsInMethodIfSupported (object=, method=, location=0x7ff8a3d50c8a "the test body") at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 #13 0x00007ff8a3d11f39 in testing::Test::Run (this=0x7b2000001080) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2474 #14 0x00007ff8a3d131dd in testing::TestInfo::Run (this=0x7b3c00001d10) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2656 #15 0x00007ff8a3d13ef7 in testing::TestCase::Run (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2774 #16 0x00007ff8a3d203a7 in testing::internal::UnitTestImpl::RunAllTests (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4649 #17 0x00007ff8a3d33e50 in HandleSehExceptionsInMethodIfSupported (object=0x7b5000000200, method=, location=0x7ff8a3d5138b "auxiliary test code (environments or event listeners)") at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2402 #18 testing::internal::HandleExceptionsInMethodIfSupported (object=, method=, location=0x7ff8a3d5138b "auxiliary test code (environments or event listeners)") at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:2438 #19 0x00007ff8a3d1fc62 in testing::UnitTest::Run (this=) at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/googletest-release-1.8.0/googletest/src/gtest.cc:4257 #20 0x00007ff8a45b570c in RUN_ALL_TESTS () at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/installed/tsan/include/gtest/gtest.h:2233 #21 0x00007ff8a45b4587 in main (argc=1, argv=0x7ffcdcd2daf8) at /home/jenkins-slave/workspace/kudu-master/0/src/kudu/util/test_main.cc:106 #22 0x00007ff892a63f45 in __libc_start_main (main=0x4c1470 , argc=3, argv=0x7ffcdcd2dae8, init=, fini=, rtld_fini=, stack_end=0x7ffcdcd2dad8) at libc-start.c:287 #23 0x00000000004c8ea8 in _start () ************************* END STACKS *************************** F0521 22:43:09.132889 363 test_main.cc:63] Maximum unit test time exceeded (870 sec) *** Check failure stack trace: *** *** Aborted at 1590100989 (unix time) try "date -d @1590100989" if you are using GNU date *** PC: @ 0x7ff892a78c37 gsignal *** SIGABRT (@0x3e800000167) received by PID 359 (TID 0x7ff88f0fe700) from PID 359; stack trace: *** @ 0x4d06bd __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1916 @ 0x4d17aa rtl_sigaction() at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:2006 @ 0x7ff89d09e330 (unknown) at ??:0 @ 0x7ff892a78c37 gsignal at ??:0 @ 0x7ff892a7c028 abort at ??:0 @ 0x4d3fa1 abort at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1777 @ 0x7ff896383412 google::logging_fail() at ??:0 @ 0x7ff896382284 google::LogMessage::SendToLog() at ??:0 @ 0x7ff896382d22 google::LogMessage::Flush() at ??:0 @ 0x7ff89638748b google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7ff8a45b4d9e kudu::CreateAndStartTimeoutThread()::$_0::operator()() at ??:0 @ 0x7ff8a45b4c54 _ZNSt3__18__invokeIZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS3_DpOS4_ at ??:0 @ 0x7ff8a45b4bc4 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0JEJEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at ??:0 @ 0x7ff8a45b495b _ZNSt3__114__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN4kuduL27CreateAndStartTimeoutThreadEvE3$_0EEEEEPvSA_ at ??:0 @ 0x4d0580 __tsan_thread_start_func at /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:960 @ 0x7ff89d096184 start_thread at ??:0 @ 0x7ff892b3fffd clone at ??:0