[==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from WriteThrottlingTest [ RUN ] WriteThrottlingTest.ThrottleWriteRpcPerSec WARNING: Logging before InitGoogleLogging() is written to STDERR I0109 00:00:38.141026 9190 external_mini_cluster.cc:845] Running /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-master /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.8.249.190:42049 --webserver_interface=127.8.249.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0109 00:00:38.173717 9196 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0109 00:00:38.173800 9196 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0109 00:00:38.174633 9196 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0109 00:00:38.174648 9196 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0109 00:00:38.174651 9196 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0109 00:00:38.174655 9196 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0109 00:00:38.175765 9196 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.8.249.190:42049 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/data/info.pb --webserver_interface=127.8.249.190 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.9.0-SNAPSHOT revision 4bacdfafb7ba64d2c328bbef217de2f7bfd540d9 build type RELEASE built by None at 08 Jan 2019 23:55:24 UTC on jenkins-slave-8qw8 build id 15957 I0109 00:00:38.176642 9196 master_main.cc:80] Initializing master server... I0109 00:00:38.176790 9196 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 262953us I0109 00:00:38.176913 9196 fs_manager.cc:263] Metadata directory not provided I0109 00:00:38.176931 9196 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0109 00:00:38.176978 9196 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0109 00:00:38.176991 9196 server_base.cc:433] Attempting to create new FS layout instead I0109 00:00:38.177770 9196 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/data/instance: uuid: "0548a7edf12f45c3b3d0dbf607d7d911" format_stamp: "Formatted at 2019-01-09 00:00:38 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:38.177950 9196 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/wal/instance: uuid: "0548a7edf12f45c3b3d0dbf607d7d911" format_stamp: "Formatted at 2019-01-09 00:00:38 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:38.179102 9196 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0109 00:00:38.179383 9196 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0109 00:00:38.179468 9196 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0109 00:00:38.180013 9196 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0109 00:00:38.180044 9196 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/data,/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/wal uuid: "0548a7edf12f45c3b3d0dbf607d7d911" format_stamp: "Formatted at 2019-01-09 00:00:38 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:38.180075 9196 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0109 00:00:38.195899 9196 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0109 00:00:38.196069 9196 master_main.cc:83] Starting Master server... I0109 00:00:38.198308 9196 rpc_server.cc:205] RPC server started. Bound to: 127.8.249.190:42049 I0109 00:00:38.198374 9196 webserver.cc:175] Starting webserver on 127.8.249.190:0 I0109 00:00:38.198385 9196 webserver.cc:186] Document root disabled I0109 00:00:38.198503 9196 webserver.cc:313] Webserver started. Bound to: http://127.8.249.190:40019/ I0109 00:00:38.198652 9196 server_base.cc:607] Dumped server information to /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/master-0/data/info.pb I0109 00:00:38.198994 9255 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0109 00:00:38.201321 9255 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911: Bootstrap starting. I0109 00:00:38.201638 9255 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911: No blocks or log segments found. Creating new log. I0109 00:00:38.201808 9255 log.cc:525] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911: Log is configured to *not* fsync() on all Append() calls I0109 00:00:38.202122 9255 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911: No bootstrap required, opened a new log I0109 00:00:38.203078 9190 external_mini_cluster.cc:907] Started /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-master as pid 9196 I0109 00:00:38.203299 9190 external_mini_cluster.cc:845] Running /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.8.249.129:0 --local_ip_for_outbound_sockets=127.8.249.129 --webserver_interface=127.8.249.129 --webserver_port=0 --tserver_master_addrs=127.8.249.190:42049 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --tablet_throttler_rpc_per_sec=100 --tablet_throttler_bytes_per_sec=0 with env {} I0109 00:00:38.203675 9255 raft_consensus.cc:340] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0548a7edf12f45c3b3d0dbf607d7d911" member_type: VOTER } I0109 00:00:38.203703 9255 raft_consensus.cc:366] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0109 00:00:38.203712 9255 raft_consensus.cc:705] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0548a7edf12f45c3b3d0dbf607d7d911, State: Initialized, Role: FOLLOWER I0109 00:00:38.203799 9255 consensus_queue.cc:229] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0548a7edf12f45c3b3d0dbf607d7d911" member_type: VOTER } I0109 00:00:38.203816 9255 raft_consensus.cc:380] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0109 00:00:38.203824 9255 raft_consensus.cc:472] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0109 00:00:38.203841 9255 raft_consensus.cc:2834] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911: Snoozing failure detection for 2.016s (starting election) I0109 00:00:38.203860 9255 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [term 0 FOLLOWER]: Advancing to term 1 I0109 00:00:38.204347 9255 raft_consensus.cc:494] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0548a7edf12f45c3b3d0dbf607d7d911" member_type: VOTER } I0109 00:00:38.204399 9255 leader_election.cc:278] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0109 00:00:38.204524 9258 raft_consensus.cc:2834] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911: Snoozing failure detection for 2.121s (election complete) I0109 00:00:38.204552 9258 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [term 1 FOLLOWER]: Leader election won for term 1 I0109 00:00:38.204620 9258 raft_consensus.cc:667] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [term 1 LEADER]: Becoming Leader. State: Replica: 0548a7edf12f45c3b3d0dbf607d7d911, State: Running, Role: LEADER I0109 00:00:38.204689 9258 consensus_queue.cc:206] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0548a7edf12f45c3b3d0dbf607d7d911" member_type: VOTER } I0109 00:00:38.204896 9255 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "0548a7edf12f45c3b3d0dbf607d7d911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0548a7edf12f45c3b3d0dbf607d7d911" member_type: VOTER } } I0109 00:00:38.204892 9258 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0548a7edf12f45c3b3d0dbf607d7d911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0548a7edf12f45c3b3d0dbf607d7d911" member_type: VOTER } } I0109 00:00:38.204926 9255 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [sys.catalog]: This master's current role is: LEADER I0109 00:00:38.204926 9258 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [sys.catalog]: This master's current role is: LEADER I0109 00:00:38.204963 9258 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0548a7edf12f45c3b3d0dbf607d7d911. Latest consensus state: current_term: 1 leader_uuid: "0548a7edf12f45c3b3d0dbf607d7d911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0548a7edf12f45c3b3d0dbf607d7d911" member_type: VOTER } } I0109 00:00:38.205341 9258 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [sys.catalog]: This master's current role is: LEADER I0109 00:00:38.205073 9255 sys_catalog.cc:424] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911 [sys.catalog]: configured and running, proceeding with master startup. I0109 00:00:38.205665 9196 master_main.cc:86] Master server successfully started. W0109 00:00:38.205809 9264 catalog_manager.cc:1094] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0109 00:00:38.205832 9264 catalog_manager.cc:584] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0109 00:00:38.211385 9262 catalog_manager.cc:1036] Loading table and tablet metadata into memory... I0109 00:00:38.211531 9262 catalog_manager.cc:1047] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0109 00:00:38.239598 9257 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0109 00:00:38.239701 9257 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0109 00:00:38.240766 9257 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0109 00:00:38.240794 9257 flags.cc:406] Enabled experimental flag: --tablet_throttler_bytes_per_sec=0 W0109 00:00:38.240800 9257 flags.cc:406] Enabled experimental flag: --tablet_throttler_rpc_per_sec=100 W0109 00:00:38.240813 9257 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129 I0109 00:00:38.242209 9257 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.8.249.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/data/info.pb --webserver_interface=127.8.249.129 --webserver_port=0 --tablet_throttler_rpc_per_sec=100 --tserver_master_addrs=127.8.249.190:42049 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.8.249.129 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 4bacdfafb7ba64d2c328bbef217de2f7bfd540d9 build type RELEASE built by None at 08 Jan 2019 23:55:24 UTC on jenkins-slave-8qw8 build id 15957 I0109 00:00:38.243690 9257 tablet_server_main.cc:85] Initializing tablet server... I0109 00:00:38.243923 9257 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 262953us I0109 00:00:38.244055 9257 fs_manager.cc:263] Metadata directory not provided I0109 00:00:38.244084 9257 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0109 00:00:38.244148 9257 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0109 00:00:38.244166 9257 server_base.cc:433] Attempting to create new FS layout instead I0109 00:00:38.245208 9257 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/data/instance: uuid: "7d45366beeba4376bee078d1ce39b5c2" format_stamp: "Formatted at 2019-01-09 00:00:38 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:38.245445 9257 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/wal/instance: uuid: "7d45366beeba4376bee078d1ce39b5c2" format_stamp: "Formatted at 2019-01-09 00:00:38 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:38.247579 9257 fs_manager.cc:503] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s I0109 00:00:38.247902 9257 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0109 00:00:38.247985 9257 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0109 00:00:38.248770 9257 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0109 00:00:38.248811 9257 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/data,/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/wal uuid: "7d45366beeba4376bee078d1ce39b5c2" format_stamp: "Formatted at 2019-01-09 00:00:38 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:38.248852 9257 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0109 00:00:38.281076 9257 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0109 00:00:38.281422 9257 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0109 00:00:38.281493 9257 tablet_server_main.cc:90] Starting tablet server... I0109 00:00:38.285995 9257 rpc_server.cc:205] RPC server started. Bound to: 127.8.249.129:41457 I0109 00:00:38.286072 9257 webserver.cc:175] Starting webserver on 127.8.249.129:0 I0109 00:00:38.286090 9257 webserver.cc:186] Document root disabled I0109 00:00:38.286262 9257 webserver.cc:313] Webserver started. Bound to: http://127.8.249.129:34857/ I0109 00:00:38.286485 9257 server_base.cc:607] Dumped server information to /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-0/data/info.pb I0109 00:00:38.286767 9257 tablet_server_main.cc:93] Tablet server successfully started. I0109 00:00:38.292192 9384 heartbeater.cc:346] Connected to a master server at 127.8.249.190:42049 I0109 00:00:38.292235 9384 heartbeater.cc:426] Registering TS with master... I0109 00:00:38.292316 9384 heartbeater.cc:475] Master 127.8.249.190:42049 requested a full tablet report, sending... I0109 00:00:38.292662 9221 ts_manager.cc:98] Registered new tserver with Master: 7d45366beeba4376bee078d1ce39b5c2 (127.8.249.129:41457) I0109 00:00:38.295548 9262 catalog_manager.cc:931] Generated new certificate authority record I0109 00:00:38.295812 9190 external_mini_cluster.cc:907] Started /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver as pid 9257 I0109 00:00:38.296070 9190 external_mini_cluster.cc:845] Running /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.8.249.130:0 --local_ip_for_outbound_sockets=127.8.249.130 --webserver_interface=127.8.249.130 --webserver_port=0 --tserver_master_addrs=127.8.249.190:42049 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --tablet_throttler_rpc_per_sec=100 --tablet_throttler_bytes_per_sec=0 with env {} I0109 00:00:38.296147 9262 catalog_manager.cc:1056] Loading token signing keys... I0109 00:00:38.298514 9262 catalog_manager.cc:4085] T 00000000000000000000000000000000 P 0548a7edf12f45c3b3d0dbf607d7d911: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0109 00:00:38.348820 9389 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0109 00:00:38.348925 9389 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0109 00:00:38.349644 9389 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0109 00:00:38.349669 9389 flags.cc:406] Enabled experimental flag: --tablet_throttler_bytes_per_sec=0 W0109 00:00:38.349675 9389 flags.cc:406] Enabled experimental flag: --tablet_throttler_rpc_per_sec=100 W0109 00:00:38.349687 9389 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130 I0109 00:00:38.350695 9389 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.8.249.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/data/info.pb --webserver_interface=127.8.249.130 --webserver_port=0 --tablet_throttler_rpc_per_sec=100 --tserver_master_addrs=127.8.249.190:42049 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.8.249.130 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 4bacdfafb7ba64d2c328bbef217de2f7bfd540d9 build type RELEASE built by None at 08 Jan 2019 23:55:24 UTC on jenkins-slave-8qw8 build id 15957 I0109 00:00:38.355739 9389 tablet_server_main.cc:85] Initializing tablet server... I0109 00:00:38.355950 9389 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 262953us I0109 00:00:38.356075 9389 fs_manager.cc:263] Metadata directory not provided I0109 00:00:38.356099 9389 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0109 00:00:38.356170 9389 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0109 00:00:38.356186 9389 server_base.cc:433] Attempting to create new FS layout instead I0109 00:00:38.357008 9389 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/data/instance: uuid: "4f067e4c90ce4e54a77d6a08b32f007d" format_stamp: "Formatted at 2019-01-09 00:00:38 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:38.357183 9389 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/wal/instance: uuid: "4f067e4c90ce4e54a77d6a08b32f007d" format_stamp: "Formatted at 2019-01-09 00:00:38 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:38.358363 9389 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0109 00:00:38.358619 9389 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0109 00:00:38.358693 9389 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0109 00:00:38.359272 9389 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0109 00:00:38.359304 9389 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/data,/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/wal uuid: "4f067e4c90ce4e54a77d6a08b32f007d" format_stamp: "Formatted at 2019-01-09 00:00:38 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:38.359405 9389 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/wal 1 data directories: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0109 00:00:38.385601 9389 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0109 00:00:38.385854 9389 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0109 00:00:38.385921 9389 tablet_server_main.cc:90] Starting tablet server... I0109 00:00:38.389546 9389 rpc_server.cc:205] RPC server started. Bound to: 127.8.249.130:40195 I0109 00:00:38.389616 9389 webserver.cc:175] Starting webserver on 127.8.249.130:0 I0109 00:00:38.389634 9389 webserver.cc:186] Document root disabled I0109 00:00:38.389771 9389 webserver.cc:313] Webserver started. Bound to: http://127.8.249.130:39375/ I0109 00:00:38.389933 9389 server_base.cc:607] Dumped server information to /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-1/data/info.pb I0109 00:00:38.390233 9389 tablet_server_main.cc:93] Tablet server successfully started. I0109 00:00:38.390812 9190 external_mini_cluster.cc:907] Started /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver as pid 9389 I0109 00:00:38.391058 9190 external_mini_cluster.cc:845] Running /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.8.249.131:0 --local_ip_for_outbound_sockets=127.8.249.131 --webserver_interface=127.8.249.131 --webserver_port=0 --tserver_master_addrs=127.8.249.190:42049 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --tablet_throttler_rpc_per_sec=100 --tablet_throttler_bytes_per_sec=0 with env {} I0109 00:00:38.394327 9508 heartbeater.cc:346] Connected to a master server at 127.8.249.190:42049 I0109 00:00:38.394382 9508 heartbeater.cc:426] Registering TS with master... I0109 00:00:38.394471 9508 heartbeater.cc:475] Master 127.8.249.190:42049 requested a full tablet report, sending... I0109 00:00:38.394718 9221 ts_manager.cc:98] Registered new tserver with Master: 4f067e4c90ce4e54a77d6a08b32f007d (127.8.249.130:40195) I0109 00:00:38.395443 9221 master_service.cc:254] Signed X509 certificate for tserver {username='slave'} at 127.8.249.130:37499 WARNING: Logging before InitGoogleLogging() is written to STDERR W0109 00:00:38.427176 9511 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0109 00:00:38.427269 9511 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0109 00:00:38.428010 9511 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0109 00:00:38.428032 9511 flags.cc:406] Enabled experimental flag: --tablet_throttler_bytes_per_sec=0 W0109 00:00:38.428036 9511 flags.cc:406] Enabled experimental flag: --tablet_throttler_rpc_per_sec=100 W0109 00:00:38.428045 9511 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131 I0109 00:00:38.429069 9511 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.8.249.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/data/info.pb --webserver_interface=127.8.249.131 --webserver_port=0 --tablet_throttler_rpc_per_sec=100 --tserver_master_addrs=127.8.249.190:42049 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.8.249.131 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 4bacdfafb7ba64d2c328bbef217de2f7bfd540d9 build type RELEASE built by None at 08 Jan 2019 23:55:24 UTC on jenkins-slave-8qw8 build id 15957 I0109 00:00:38.430011 9511 tablet_server_main.cc:85] Initializing tablet server... I0109 00:00:38.430186 9511 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 262953us I0109 00:00:38.430271 9511 fs_manager.cc:263] Metadata directory not provided I0109 00:00:38.430290 9511 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0109 00:00:38.430339 9511 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0109 00:00:38.430362 9511 server_base.cc:433] Attempting to create new FS layout instead I0109 00:00:38.431147 9511 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/data/instance: uuid: "b00fc96538aa4e79ba03e64714dc21ef" format_stamp: "Formatted at 2019-01-09 00:00:38 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:38.431301 9511 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/wal/instance: uuid: "b00fc96538aa4e79ba03e64714dc21ef" format_stamp: "Formatted at 2019-01-09 00:00:38 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:38.432488 9511 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s I0109 00:00:38.432723 9511 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0109 00:00:38.432795 9511 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0109 00:00:38.433311 9511 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0109 00:00:38.433341 9511 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/data,/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/wal uuid: "b00fc96538aa4e79ba03e64714dc21ef" format_stamp: "Formatted at 2019-01-09 00:00:38 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:38.433380 9511 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/wal 1 data directories: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0109 00:00:38.489442 9511 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0109 00:00:38.489766 9511 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0109 00:00:38.489826 9511 tablet_server_main.cc:90] Starting tablet server... I0109 00:00:38.493372 9511 rpc_server.cc:205] RPC server started. Bound to: 127.8.249.131:37081 I0109 00:00:38.493434 9511 webserver.cc:175] Starting webserver on 127.8.249.131:0 I0109 00:00:38.493448 9511 webserver.cc:186] Document root disabled I0109 00:00:38.493582 9511 webserver.cc:313] Webserver started. Bound to: http://127.8.249.131:36627/ I0109 00:00:38.493737 9511 server_base.cc:607] Dumped server information to /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190/minicluster-data/ts-2/data/info.pb I0109 00:00:38.494071 9511 tablet_server_main.cc:93] Tablet server successfully started. I0109 00:00:38.495635 9190 external_mini_cluster.cc:907] Started /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver as pid 9511 I0109 00:00:38.498435 9630 heartbeater.cc:346] Connected to a master server at 127.8.249.190:42049 I0109 00:00:38.498476 9630 heartbeater.cc:426] Registering TS with master... I0109 00:00:38.498551 9630 heartbeater.cc:475] Master 127.8.249.190:42049 requested a full tablet report, sending... I0109 00:00:38.498803 9221 ts_manager.cc:98] Registered new tserver with Master: b00fc96538aa4e79ba03e64714dc21ef (127.8.249.131:37081) I0109 00:00:38.499563 9221 master_service.cc:254] Signed X509 certificate for tserver {username='slave'} at 127.8.249.131:51231 I0109 00:00:38.501885 9190 external_mini_cluster.cc:535] 3 TS(s) registered with all masters I0109 00:00:38.507771 9221 catalog_manager.cc:1376] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37778: name: "write-throttling-test-tbl" schema { columns { name: "key" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0109 00:00:38.512034 9567 tablet_service.cc:773] Processing CreateTablet for tablet ffb707a7852b408db729505930710e7d (table=write-throttling-test-tbl [id=18253cd8943e4bcf97d60c34ea5e3ed1]), partition=RANGE (key) PARTITION UNBOUNDED I0109 00:00:38.512142 9567 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet ffb707a7852b408db729505930710e7d. 1 dirs total, 0 dirs full, 0 dirs failed I0109 00:00:38.513976 9567 ts_tablet_manager.cc:1173] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef: Registered tablet (data state: TABLET_DATA_READY) I0109 00:00:38.514084 9642 ts_tablet_manager.cc:1028] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef: Bootstrapping tablet I0109 00:00:38.514153 9642 tablet_bootstrap.cc:438] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef: Bootstrap starting. I0109 00:00:38.514520 9642 tablet_bootstrap.cc:592] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef: No blocks or log segments found. Creating new log. I0109 00:00:38.514742 9642 log.cc:525] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef: Log is configured to *not* fsync() on all Append() calls I0109 00:00:38.515087 9642 tablet_bootstrap.cc:438] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef: No bootstrap required, opened a new log I0109 00:00:38.515127 9642 ts_tablet_manager.cc:1045] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0109 00:00:38.516444 9642 raft_consensus.cc:340] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00fc96538aa4e79ba03e64714dc21ef" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37081 } } I0109 00:00:38.516474 9642 raft_consensus.cc:366] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0109 00:00:38.516484 9642 raft_consensus.cc:705] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00fc96538aa4e79ba03e64714dc21ef, State: Initialized, Role: FOLLOWER I0109 00:00:38.516585 9642 consensus_queue.cc:229] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00fc96538aa4e79ba03e64714dc21ef" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37081 } } I0109 00:00:38.516611 9642 raft_consensus.cc:380] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0109 00:00:38.516624 9642 raft_consensus.cc:472] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0109 00:00:38.516649 9642 raft_consensus.cc:2834] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef: Snoozing failure detection for 1.574s (starting election) I0109 00:00:38.516665 9642 raft_consensus.cc:2886] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef [term 0 FOLLOWER]: Advancing to term 1 I0109 00:00:38.580103 9642 raft_consensus.cc:494] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00fc96538aa4e79ba03e64714dc21ef" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37081 } } I0109 00:00:38.580169 9642 leader_election.cc:278] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0109 00:00:38.580309 9644 raft_consensus.cc:2834] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef: Snoozing failure detection for 1.877s (election complete) I0109 00:00:38.580338 9644 raft_consensus.cc:2640] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef [term 1 FOLLOWER]: Leader election won for term 1 I0109 00:00:38.580394 9630 heartbeater.cc:467] Master 127.8.249.190:42049 was elected leader, sending a full tablet report... I0109 00:00:38.580464 9642 ts_tablet_manager.cc:1073] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef: Time spent starting tablet: real 0.065s user 0.003s sys 0.000s I0109 00:00:38.580579 9644 raft_consensus.cc:667] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef [term 1 LEADER]: Becoming Leader. State: Replica: b00fc96538aa4e79ba03e64714dc21ef, State: Running, Role: LEADER I0109 00:00:38.580662 9644 consensus_queue.cc:206] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00fc96538aa4e79ba03e64714dc21ef" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37081 } } I0109 00:00:38.581320 9221 catalog_manager.cc:3781] T ffb707a7852b408db729505930710e7d P b00fc96538aa4e79ba03e64714dc21ef reported cstate change: term changed from 0 to 1, leader changed from to b00fc96538aa4e79ba03e64714dc21ef (127.8.249.131). New cstate: current_term: 1 leader_uuid: "b00fc96538aa4e79ba03e64714dc21ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b00fc96538aa4e79ba03e64714dc21ef" member_type: VOTER last_known_addr { host: "127.8.249.131" port: 37081 } health_report { overall_health: HEALTHY } } } I0109 00:00:39.293716 9221 master_service.cc:254] Signed X509 certificate for tserver {username='slave'} at 127.8.249.129:32851 I0109 00:00:39.396006 9508 heartbeater.cc:467] Master 127.8.249.190:42049 was elected leader, sending a full tablet report... I0109 00:00:39.419872 9190 write_throttling-itest.cc:108] Iteration 0 qps: 122.07 /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/write_throttling-itest.cc:109: Failure Expected: (qps) <= (TARGET_QPS * 1.2f), actual: 122.07 vs 120 I0109 00:00:39.420045 9190 external_mini_cluster-itest-base.cc:80] Found fatal failure I0109 00:00:39.420078 9190 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 7d45366beeba4376bee078d1ce39b5c2 and pid 9257 ************************ BEGIN STACKS ************************** 0x00007f64b9d24d4d in ?? () Id Target Id Frame * 1 process 9257 "kudu-tserver" 0x00007f64b9d24d4d in ?? () Thread 1 (process 9257): #0 0x00007f64b9d24d4d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0109 00:00:39.556336 9190 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID 4f067e4c90ce4e54a77d6a08b32f007d and pid 9389 ************************ BEGIN STACKS ************************** 0x00007f2a4f3c1d4d in ?? () Id Target Id Frame * 1 process 9389 "kudu-tserver" 0x00007f2a4f3c1d4d in ?? () Thread 1 (process 9389): #0 0x00007f2a4f3c1d4d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0109 00:00:39.691879 9190 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID b00fc96538aa4e79ba03e64714dc21ef and pid 9511 ************************ BEGIN STACKS ************************** 0x00007fd275bcbd4d in ?? () Id Target Id Frame * 1 process 9511 "kudu-tserver" 0x00007fd275bcbd4d in ?? () Thread 1 (process 9511): #0 0x00007fd275bcbd4d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0109 00:00:39.824148 9190 external_mini_cluster.cc:1043] Killing /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver with pid 9257 W0109 00:00:39.827116 9205 connection.cc:512] server connection from 127.8.249.129:32851 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.129:32851): Connection reset by peer (error 104) I0109 00:00:39.827162 9190 external_mini_cluster.cc:1043] Killing /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver with pid 9389 W0109 00:00:39.829526 9203 connection.cc:512] server connection from 127.8.249.130:37499 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.130:37499): Connection reset by peer (error 104) I0109 00:00:39.829568 9190 external_mini_cluster.cc:1043] Killing /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver with pid 9511 W0109 00:00:39.831918 9205 connection.cc:512] server connection from 127.8.249.131:51231 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.131:51231): Connection reset by peer (error 104) W0109 00:00:39.831965 9637 connection.cc:512] client connection to 127.8.249.131:37081 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.131:37081): Connection reset by peer (error 104) W0109 00:00:39.832015 9205 connection.cc:512] client connection to 127.8.249.131:37081 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.131:37081): Connection reset by peer (error 104) I0109 00:00:39.832057 9190 external_mini_cluster.cc:1043] Killing /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-master with pid 9196 W0109 00:00:39.834695 9636 connection.cc:512] client connection to 127.8.249.190:42049 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.190:42049): Connection reset by peer (error 104) W0109 00:00:39.834704 9195 connection.cc:512] client connection to 127.8.249.190:42049 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.190:42049): Connection reset by peer (error 104) I0109 00:00:39.835383 9190 test_util.cc:132] ----------------------------------------------- I0109 00:00:39.835403 9190 test_util.cc:133] Had fatal failures, leaving test files at /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteRpcPerSec.1546992038132538-9190 [ FAILED ] WriteThrottlingTest.ThrottleWriteRpcPerSec (1699 ms) [ RUN ] WriteThrottlingTest.ThrottleWriteBytesPerSec I0109 00:00:39.836261 9190 external_mini_cluster.cc:845] Running /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-master /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.8.249.190:40449 --webserver_interface=127.8.249.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0109 00:00:39.869047 9676 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0109 00:00:39.869127 9676 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0109 00:00:39.869994 9676 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0109 00:00:39.870010 9676 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0109 00:00:39.870014 9676 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0109 00:00:39.870018 9676 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0109 00:00:39.871137 9676 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.8.249.190:40449 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/data/info.pb --webserver_interface=127.8.249.190 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.9.0-SNAPSHOT revision 4bacdfafb7ba64d2c328bbef217de2f7bfd540d9 build type RELEASE built by None at 08 Jan 2019 23:55:24 UTC on jenkins-slave-8qw8 build id 15957 I0109 00:00:39.872041 9676 master_main.cc:80] Initializing master server... I0109 00:00:39.872264 9676 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 263453us I0109 00:00:39.872422 9676 fs_manager.cc:263] Metadata directory not provided I0109 00:00:39.872443 9676 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0109 00:00:39.872505 9676 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0109 00:00:39.872524 9676 server_base.cc:433] Attempting to create new FS layout instead I0109 00:00:39.873332 9676 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/data/instance: uuid: "ed907d436e2e43f2b59417350e0154da" format_stamp: "Formatted at 2019-01-09 00:00:39 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:39.873523 9676 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/wal/instance: uuid: "ed907d436e2e43f2b59417350e0154da" format_stamp: "Formatted at 2019-01-09 00:00:39 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:39.874644 9676 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I0109 00:00:39.874868 9676 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0109 00:00:39.874945 9676 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0109 00:00:39.875491 9676 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0109 00:00:39.875524 9676 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/data,/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/wal uuid: "ed907d436e2e43f2b59417350e0154da" format_stamp: "Formatted at 2019-01-09 00:00:39 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:39.875550 9676 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/wal 1 data directories: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0109 00:00:39.903506 9676 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0109 00:00:39.903667 9676 master_main.cc:83] Starting Master server... I0109 00:00:39.906229 9676 rpc_server.cc:205] RPC server started. Bound to: 127.8.249.190:40449 I0109 00:00:39.906292 9676 webserver.cc:175] Starting webserver on 127.8.249.190:0 I0109 00:00:39.906319 9676 webserver.cc:186] Document root disabled I0109 00:00:39.906492 9676 webserver.cc:313] Webserver started. Bound to: http://127.8.249.190:34877/ I0109 00:00:39.906672 9676 server_base.cc:607] Dumped server information to /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/master-0/data/info.pb I0109 00:00:39.907037 9735 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0109 00:00:39.908470 9190 external_mini_cluster.cc:907] Started /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-master as pid 9676 I0109 00:00:39.908673 9190 external_mini_cluster.cc:845] Running /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.8.249.129:0 --local_ip_for_outbound_sockets=127.8.249.129 --webserver_interface=127.8.249.129 --webserver_port=0 --tserver_master_addrs=127.8.249.190:40449 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --tablet_throttler_rpc_per_sec=0 --tablet_throttler_bytes_per_sec=300000 with env {} I0109 00:00:39.909538 9735 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da: Bootstrap starting. I0109 00:00:39.909878 9735 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da: No blocks or log segments found. Creating new log. I0109 00:00:39.910081 9735 log.cc:525] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da: Log is configured to *not* fsync() on all Append() calls I0109 00:00:39.910715 9735 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da: No bootstrap required, opened a new log I0109 00:00:39.913015 9735 raft_consensus.cc:340] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed907d436e2e43f2b59417350e0154da" member_type: VOTER } I0109 00:00:39.913054 9735 raft_consensus.cc:366] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0109 00:00:39.913069 9735 raft_consensus.cc:705] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed907d436e2e43f2b59417350e0154da, State: Initialized, Role: FOLLOWER I0109 00:00:39.913166 9735 consensus_queue.cc:229] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed907d436e2e43f2b59417350e0154da" member_type: VOTER } I0109 00:00:39.913192 9735 raft_consensus.cc:380] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0109 00:00:39.913203 9735 raft_consensus.cc:472] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0109 00:00:39.913224 9735 raft_consensus.cc:2834] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da: Snoozing failure detection for 1.534s (starting election) I0109 00:00:39.913239 9735 raft_consensus.cc:2886] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [term 0 FOLLOWER]: Advancing to term 1 I0109 00:00:39.913941 9735 raft_consensus.cc:494] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed907d436e2e43f2b59417350e0154da" member_type: VOTER } I0109 00:00:39.913993 9735 leader_election.cc:278] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0109 00:00:39.914117 9738 raft_consensus.cc:2834] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da: Snoozing failure detection for 1.584s (election complete) I0109 00:00:39.914144 9738 raft_consensus.cc:2640] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [term 1 FOLLOWER]: Leader election won for term 1 I0109 00:00:39.914213 9738 raft_consensus.cc:667] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [term 1 LEADER]: Becoming Leader. State: Replica: ed907d436e2e43f2b59417350e0154da, State: Running, Role: LEADER I0109 00:00:39.914278 9738 consensus_queue.cc:206] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed907d436e2e43f2b59417350e0154da" member_type: VOTER } I0109 00:00:39.914503 9735 sys_catalog.cc:337] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "ed907d436e2e43f2b59417350e0154da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed907d436e2e43f2b59417350e0154da" member_type: VOTER } } I0109 00:00:39.914537 9735 sys_catalog.cc:340] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [sys.catalog]: This master's current role is: LEADER I0109 00:00:39.914553 9739 sys_catalog.cc:337] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed907d436e2e43f2b59417350e0154da. Latest consensus state: current_term: 1 leader_uuid: "ed907d436e2e43f2b59417350e0154da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed907d436e2e43f2b59417350e0154da" member_type: VOTER } } I0109 00:00:39.914563 9739 sys_catalog.cc:340] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [sys.catalog]: This master's current role is: LEADER I0109 00:00:39.914671 9735 sys_catalog.cc:424] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [sys.catalog]: configured and running, proceeding with master startup. I0109 00:00:39.914503 9738 sys_catalog.cc:337] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed907d436e2e43f2b59417350e0154da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed907d436e2e43f2b59417350e0154da" member_type: VOTER } } I0109 00:00:39.914968 9676 master_main.cc:86] Master server successfully started. I0109 00:00:39.915004 9738 sys_catalog.cc:340] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da [sys.catalog]: This master's current role is: LEADER W0109 00:00:39.915242 9743 catalog_manager.cc:1094] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0109 00:00:39.915261 9743 catalog_manager.cc:584] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0109 00:00:39.919416 9742 catalog_manager.cc:1036] Loading table and tablet metadata into memory... I0109 00:00:39.919555 9742 catalog_manager.cc:1047] Initializing Kudu internal certificate authority... I0109 00:00:39.939020 9742 catalog_manager.cc:931] Generated new certificate authority record I0109 00:00:39.939587 9742 catalog_manager.cc:1056] Loading token signing keys... WARNING: Logging before InitGoogleLogging() is written to STDERR W0109 00:00:39.943733 9736 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0109 00:00:39.943842 9736 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0109 00:00:39.945446 9736 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0109 00:00:39.945479 9736 flags.cc:406] Enabled experimental flag: --tablet_throttler_bytes_per_sec=300000 W0109 00:00:39.945485 9736 flags.cc:406] Enabled experimental flag: --tablet_throttler_rpc_per_sec=0 W0109 00:00:39.945499 9736 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.129 I0109 00:00:39.946971 9736 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.8.249.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/data/info.pb --webserver_interface=127.8.249.129 --webserver_port=0 --tablet_throttler_bytes_per_sec=300000 --tserver_master_addrs=127.8.249.190:40449 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.8.249.129 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 4bacdfafb7ba64d2c328bbef217de2f7bfd540d9 build type RELEASE built by None at 08 Jan 2019 23:55:24 UTC on jenkins-slave-8qw8 build id 15957 I0109 00:00:39.947439 9742 catalog_manager.cc:4085] T 00000000000000000000000000000000 P ed907d436e2e43f2b59417350e0154da: Generated new TSK 0 I0109 00:00:39.948514 9736 tablet_server_main.cc:85] Initializing tablet server... I0109 00:00:39.948753 9736 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 263453us I0109 00:00:39.948881 9736 fs_manager.cc:263] Metadata directory not provided I0109 00:00:39.948910 9736 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0109 00:00:39.948976 9736 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0109 00:00:39.948997 9736 server_base.cc:433] Attempting to create new FS layout instead I0109 00:00:39.950054 9736 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/data/instance: uuid: "d6c6f89fdac441a7b25290a3391f02a9" format_stamp: "Formatted at 2019-01-09 00:00:39 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:39.950271 9736 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/wal/instance: uuid: "d6c6f89fdac441a7b25290a3391f02a9" format_stamp: "Formatted at 2019-01-09 00:00:39 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:39.951997 9736 fs_manager.cc:503] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s I0109 00:00:39.952330 9736 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0109 00:00:39.952437 9736 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0109 00:00:39.953197 9736 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0109 00:00:39.953238 9736 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/data,/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/wal uuid: "d6c6f89fdac441a7b25290a3391f02a9" format_stamp: "Formatted at 2019-01-09 00:00:39 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:39.953277 9736 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/wal 1 data directories: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0109 00:00:39.987493 9736 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0109 00:00:39.987828 9736 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0109 00:00:39.987897 9736 tablet_server_main.cc:90] Starting tablet server... I0109 00:00:39.991559 9736 rpc_server.cc:205] RPC server started. Bound to: 127.8.249.129:46143 I0109 00:00:39.991619 9736 webserver.cc:175] Starting webserver on 127.8.249.129:0 I0109 00:00:39.991636 9736 webserver.cc:186] Document root disabled I0109 00:00:39.991811 9736 webserver.cc:313] Webserver started. Bound to: http://127.8.249.129:33977/ I0109 00:00:39.991986 9736 server_base.cc:607] Dumped server information to /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-0/data/info.pb I0109 00:00:39.992242 9736 tablet_server_main.cc:93] Tablet server successfully started. I0109 00:00:39.997151 9865 heartbeater.cc:346] Connected to a master server at 127.8.249.190:40449 I0109 00:00:39.997186 9865 heartbeater.cc:426] Registering TS with master... I0109 00:00:39.997258 9865 heartbeater.cc:475] Master 127.8.249.190:40449 requested a full tablet report, sending... I0109 00:00:39.997609 9702 ts_manager.cc:98] Registered new tserver with Master: d6c6f89fdac441a7b25290a3391f02a9 (127.8.249.129:46143) I0109 00:00:39.998349 9702 master_service.cc:254] Signed X509 certificate for tserver {username='slave'} at 127.8.249.129:59231 I0109 00:00:40.001969 9190 external_mini_cluster.cc:907] Started /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver as pid 9736 I0109 00:00:40.002218 9190 external_mini_cluster.cc:845] Running /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.8.249.130:0 --local_ip_for_outbound_sockets=127.8.249.130 --webserver_interface=127.8.249.130 --webserver_port=0 --tserver_master_addrs=127.8.249.190:40449 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --tablet_throttler_rpc_per_sec=0 --tablet_throttler_bytes_per_sec=300000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0109 00:00:40.047297 9869 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0109 00:00:40.047461 9869 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0109 00:00:40.048544 9869 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0109 00:00:40.048575 9869 flags.cc:406] Enabled experimental flag: --tablet_throttler_bytes_per_sec=300000 W0109 00:00:40.048581 9869 flags.cc:406] Enabled experimental flag: --tablet_throttler_rpc_per_sec=0 W0109 00:00:40.048593 9869 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.130 I0109 00:00:40.050052 9869 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.8.249.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/data/info.pb --webserver_interface=127.8.249.130 --webserver_port=0 --tablet_throttler_bytes_per_sec=300000 --tserver_master_addrs=127.8.249.190:40449 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.8.249.130 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 4bacdfafb7ba64d2c328bbef217de2f7bfd540d9 build type RELEASE built by None at 08 Jan 2019 23:55:24 UTC on jenkins-slave-8qw8 build id 15957 I0109 00:00:40.051564 9869 tablet_server_main.cc:85] Initializing tablet server... I0109 00:00:40.051800 9869 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 263953us I0109 00:00:40.051926 9869 fs_manager.cc:263] Metadata directory not provided I0109 00:00:40.051954 9869 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0109 00:00:40.052026 9869 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0109 00:00:40.052044 9869 server_base.cc:433] Attempting to create new FS layout instead I0109 00:00:40.053131 9869 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/data/instance: uuid: "d34a5153492c4c0cb16d414c313b6c54" format_stamp: "Formatted at 2019-01-09 00:00:40 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:40.053349 9869 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/wal/instance: uuid: "d34a5153492c4c0cb16d414c313b6c54" format_stamp: "Formatted at 2019-01-09 00:00:40 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:40.055011 9869 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s I0109 00:00:40.055260 9869 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0109 00:00:40.055375 9869 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0109 00:00:40.055918 9869 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0109 00:00:40.055956 9869 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/data,/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/wal uuid: "d34a5153492c4c0cb16d414c313b6c54" format_stamp: "Formatted at 2019-01-09 00:00:40 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:40.055987 9869 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/wal 1 data directories: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0109 00:00:40.088225 9869 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0109 00:00:40.088496 9869 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0109 00:00:40.088557 9869 tablet_server_main.cc:90] Starting tablet server... I0109 00:00:40.092154 9869 rpc_server.cc:205] RPC server started. Bound to: 127.8.249.130:36733 I0109 00:00:40.092216 9869 webserver.cc:175] Starting webserver on 127.8.249.130:0 I0109 00:00:40.092231 9869 webserver.cc:186] Document root disabled I0109 00:00:40.092391 9869 webserver.cc:313] Webserver started. Bound to: http://127.8.249.130:40709/ I0109 00:00:40.092556 9869 server_base.cc:607] Dumped server information to /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-1/data/info.pb I0109 00:00:40.092818 9869 tablet_server_main.cc:93] Tablet server successfully started. I0109 00:00:40.096900 9988 heartbeater.cc:346] Connected to a master server at 127.8.249.190:40449 I0109 00:00:40.096935 9988 heartbeater.cc:426] Registering TS with master... I0109 00:00:40.096989 9988 heartbeater.cc:475] Master 127.8.249.190:40449 requested a full tablet report, sending... I0109 00:00:40.097209 9702 ts_manager.cc:98] Registered new tserver with Master: d34a5153492c4c0cb16d414c313b6c54 (127.8.249.130:36733) I0109 00:00:40.097668 9702 master_service.cc:254] Signed X509 certificate for tserver {username='slave'} at 127.8.249.130:42213 I0109 00:00:40.098856 9190 external_mini_cluster.cc:907] Started /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver as pid 9869 I0109 00:00:40.099036 9190 external_mini_cluster.cc:845] Running /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.8.249.131:0 --local_ip_for_outbound_sockets=127.8.249.131 --webserver_interface=127.8.249.131 --webserver_port=0 --tserver_master_addrs=127.8.249.190:40449 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --tablet_throttler_rpc_per_sec=0 --tablet_throttler_bytes_per_sec=300000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0109 00:00:40.124843 9991 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0109 00:00:40.124930 9991 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0109 00:00:40.125627 9991 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0109 00:00:40.125646 9991 flags.cc:406] Enabled experimental flag: --tablet_throttler_bytes_per_sec=300000 W0109 00:00:40.125649 9991 flags.cc:406] Enabled experimental flag: --tablet_throttler_rpc_per_sec=0 W0109 00:00:40.125658 9991 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.249.131 I0109 00:00:40.126632 9991 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.8.249.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/data/info.pb --webserver_interface=127.8.249.131 --webserver_port=0 --tablet_throttler_bytes_per_sec=300000 --tserver_master_addrs=127.8.249.190:40449 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.8.249.131 --log_dir=/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 4bacdfafb7ba64d2c328bbef217de2f7bfd540d9 build type RELEASE built by None at 08 Jan 2019 23:55:24 UTC on jenkins-slave-8qw8 build id 15957 I0109 00:00:40.127658 9991 tablet_server_main.cc:85] Initializing tablet server... I0109 00:00:40.127830 9991 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 263953us I0109 00:00:40.127918 9991 fs_manager.cc:263] Metadata directory not provided I0109 00:00:40.127936 9991 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0109 00:00:40.127982 9991 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0109 00:00:40.128003 9991 server_base.cc:433] Attempting to create new FS layout instead I0109 00:00:40.128751 9991 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/data/instance: uuid: "3f89dcd03ff2402388780d3295ec38f4" format_stamp: "Formatted at 2019-01-09 00:00:40 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:40.128907 9991 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/wal/instance: uuid: "3f89dcd03ff2402388780d3295ec38f4" format_stamp: "Formatted at 2019-01-09 00:00:40 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:40.130002 9991 fs_manager.cc:503] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I0109 00:00:40.130206 9991 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0109 00:00:40.130266 9991 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0109 00:00:40.130822 9991 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0109 00:00:40.130858 9991 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/data,/tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/wal uuid: "3f89dcd03ff2402388780d3295ec38f4" format_stamp: "Formatted at 2019-01-09 00:00:40 on dist-test-slave-dist-test-slave-7zz1" I0109 00:00:40.130908 9991 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/wal 1 data directories: /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0109 00:00:40.146239 9991 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0109 00:00:40.146493 9991 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0109 00:00:40.146551 9991 tablet_server_main.cc:90] Starting tablet server... I0109 00:00:40.150043 9991 rpc_server.cc:205] RPC server started. Bound to: 127.8.249.131:39187 I0109 00:00:40.150094 9991 webserver.cc:175] Starting webserver on 127.8.249.131:0 I0109 00:00:40.150110 9991 webserver.cc:186] Document root disabled I0109 00:00:40.150226 9991 webserver.cc:313] Webserver started. Bound to: http://127.8.249.131:41349/ I0109 00:00:40.150348 9991 server_base.cc:607] Dumped server information to /tmp/dist-test-taskLZMRGm/test-tmp/write_throttling-itest.0.WriteThrottlingTest.ThrottleWriteBytesPerSec.1546992038132538-9190/minicluster-data/ts-2/data/info.pb I0109 00:00:40.150630 9991 tablet_server_main.cc:93] Tablet server successfully started. I0109 00:00:40.151113 9190 external_mini_cluster.cc:907] Started /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver as pid 9991 I0109 00:00:40.155362 10110 heartbeater.cc:346] Connected to a master server at 127.8.249.190:40449 I0109 00:00:40.155411 10110 heartbeater.cc:426] Registering TS with master... I0109 00:00:40.155495 10110 heartbeater.cc:475] Master 127.8.249.190:40449 requested a full tablet report, sending... I0109 00:00:40.155720 9702 ts_manager.cc:98] Registered new tserver with Master: 3f89dcd03ff2402388780d3295ec38f4 (127.8.249.131:39187) I0109 00:00:40.156142 9702 master_service.cc:254] Signed X509 certificate for tserver {username='slave'} at 127.8.249.131:41291 I0109 00:00:40.156687 9190 external_mini_cluster.cc:535] 3 TS(s) registered with all masters I0109 00:00:40.161722 9702 catalog_manager.cc:1376] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40490: name: "write-throttling-test-tbl" schema { columns { name: "key" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0109 00:00:40.166419 9925 tablet_service.cc:773] Processing CreateTablet for tablet cd7ddd76ece949408477c17eb5ea4b7d (table=write-throttling-test-tbl [id=5ef3528265e84b8c827228604fbca35b]), partition=RANGE (key) PARTITION UNBOUNDED I0109 00:00:40.166548 9925 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet cd7ddd76ece949408477c17eb5ea4b7d. 1 dirs total, 0 dirs full, 0 dirs failed I0109 00:00:40.168386 9925 ts_tablet_manager.cc:1173] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54: Registered tablet (data state: TABLET_DATA_READY) I0109 00:00:40.168515 10122 ts_tablet_manager.cc:1028] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54: Bootstrapping tablet I0109 00:00:40.168586 10122 tablet_bootstrap.cc:438] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54: Bootstrap starting. I0109 00:00:40.168902 10122 tablet_bootstrap.cc:592] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54: No blocks or log segments found. Creating new log. I0109 00:00:40.169100 10122 log.cc:525] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54: Log is configured to *not* fsync() on all Append() calls I0109 00:00:40.169447 10122 tablet_bootstrap.cc:438] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54: No bootstrap required, opened a new log I0109 00:00:40.169488 10122 ts_tablet_manager.cc:1045] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0109 00:00:40.170764 10122 raft_consensus.cc:340] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34a5153492c4c0cb16d414c313b6c54" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36733 } } I0109 00:00:40.170795 10122 raft_consensus.cc:366] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0109 00:00:40.170804 10122 raft_consensus.cc:705] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34a5153492c4c0cb16d414c313b6c54, State: Initialized, Role: FOLLOWER I0109 00:00:40.170892 10122 consensus_queue.cc:229] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34a5153492c4c0cb16d414c313b6c54" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36733 } } I0109 00:00:40.170914 10122 raft_consensus.cc:380] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0109 00:00:40.170924 10122 raft_consensus.cc:472] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0109 00:00:40.170940 10122 raft_consensus.cc:2834] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54: Snoozing failure detection for 1.742s (starting election) I0109 00:00:40.170959 10122 raft_consensus.cc:2886] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 [term 0 FOLLOWER]: Advancing to term 1 I0109 00:00:40.171515 10122 raft_consensus.cc:494] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34a5153492c4c0cb16d414c313b6c54" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36733 } } I0109 00:00:40.171567 10122 leader_election.cc:278] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0109 00:00:40.171759 9988 heartbeater.cc:467] Master 127.8.249.190:40449 was elected leader, sending a full tablet report... I0109 00:00:40.171772 10122 ts_tablet_manager.cc:1073] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s I0109 00:00:40.171828 10124 raft_consensus.cc:2834] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54: Snoozing failure detection for 2.093s (election complete) I0109 00:00:40.171839 10124 raft_consensus.cc:2640] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 [term 1 FOLLOWER]: Leader election won for term 1 I0109 00:00:40.171901 10124 raft_consensus.cc:667] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 [term 1 LEADER]: Becoming Leader. State: Replica: d34a5153492c4c0cb16d414c313b6c54, State: Running, Role: LEADER I0109 00:00:40.171977 10124 consensus_queue.cc:206] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34a5153492c4c0cb16d414c313b6c54" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36733 } } I0109 00:00:40.172667 9702 catalog_manager.cc:3781] T cd7ddd76ece949408477c17eb5ea4b7d P d34a5153492c4c0cb16d414c313b6c54 reported cstate change: term changed from 0 to 1, leader changed from to d34a5153492c4c0cb16d414c313b6c54 (127.8.249.130). New cstate: current_term: 1 leader_uuid: "d34a5153492c4c0cb16d414c313b6c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34a5153492c4c0cb16d414c313b6c54" member_type: VOTER last_known_addr { host: "127.8.249.130" port: 36733 } health_report { overall_health: HEALTHY } } } I0109 00:00:40.998948 9865 heartbeater.cc:467] Master 127.8.249.190:40449 was elected leader, sending a full tablet report... I0109 00:00:41.156711 10110 heartbeater.cc:467] Master 127.8.249.190:40449 was elected leader, sending a full tablet report... I0109 00:00:41.282001 9190 write_throttling-itest.cc:141] Iteration 0 qps: 90.3853 271156 byte/s I0109 00:00:42.380502 9190 write_throttling-itest.cc:141] Iteration 1 qps: 91.0374 273112 byte/s I0109 00:00:43.474548 9190 write_throttling-itest.cc:141] Iteration 2 qps: 91.4084 274225 byte/s I0109 00:00:43.474606 9190 external_mini_cluster.cc:1043] Killing /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver with pid 9736 W0109 00:00:43.477038 9684 connection.cc:512] server connection from 127.8.249.129:59231 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.129:59231): Connection reset by peer (error 104) I0109 00:00:43.477092 9190 external_mini_cluster.cc:1043] Killing /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver with pid 9869 W0109 00:00:43.479565 10116 connection.cc:512] client connection to 127.8.249.130:36733 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.130:36733): Connection reset by peer (error 104) W0109 00:00:43.479588 9685 connection.cc:512] server connection from 127.8.249.130:42213 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.130:42213): Connection reset by peer (error 104) W0109 00:00:43.479598 9684 connection.cc:512] client connection to 127.8.249.130:36733 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.130:36733): Connection reset by peer (error 104) I0109 00:00:43.479668 9190 external_mini_cluster.cc:1043] Killing /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-tserver with pid 9991 W0109 00:00:43.482622 9684 connection.cc:512] server connection from 127.8.249.131:41291 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.131:41291): Connection reset by peer (error 104) I0109 00:00:43.482694 9190 external_mini_cluster.cc:1043] Killing /tmp/dist-test-taskLZMRGm/build/release/bin/kudu-master with pid 9676 W0109 00:00:43.485061 10117 connection.cc:512] client connection to 127.8.249.190:40449 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.190:40449): Connection reset by peer (error 104) W0109 00:00:43.485065 9675 connection.cc:512] client connection to 127.8.249.190:40449 recv error: Network error: failed to read from TLS socket (remote: 127.8.249.190:40449): Connection reset by peer (error 104) [ OK ] WriteThrottlingTest.ThrottleWriteBytesPerSec (3652 ms) [----------] 2 tests from WriteThrottlingTest (5351 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (5351 ms total) [ PASSED ] 1 test. [ FAILED ] 1 test, listed below: [ FAILED ] WriteThrottlingTest.ThrottleWriteRpcPerSec 1 FAILED TEST