JUnit version 4.12 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/tmp/dist-test-taskc0vaMl/java/build/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/dist-test-taskc0vaMl/java/build/jars/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/dist-test-taskc0vaMl/java/build/jars/logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 00:27:22.878 [INFO - main] (RandomUtils.java:46) Using random seed: 1550622442876 .00:27:22.909 [INFO - main] (KuduTestHarness.java:140) Creating a new MiniKuduCluster... 00:27:22.917 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskc0vaMl/test-tmp 00:27:22.931 [INFO - main] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin 00:27:22.943 [INFO - main] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskc0vaMl/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb] 00:27:23.197 [DEBUG - main] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 3 enable_kerberos: false cluster_root: "/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684" hms_mode: NONE mini_kdc_options { } } 00:27:23.235 [DEBUG - main] (MiniKuduCluster.java:165) Response: 00:27:23.236 [DEBUG - main] (MiniKuduCluster.java:155) Request: start_cluster { } 00:27:23.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.259332 22607 external_mini_cluster.cc:933] Running /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master 00:27:23.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master 00:27:23.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/wal 00:27:23.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/data 00:27:23.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 00:27:23.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 00:27:23.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 00:27:23.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 00:27:23.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.22.19.254:44849 00:27:23.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.22.19.254 00:27:23.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:23.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 00:27:23.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:23.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:23.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:23.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:23.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 00:27:23.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:23.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/logs 00:27:23.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/data/info.pb 00:27:23.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:23.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 00:27:23.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 00:27:23.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 00:27:23.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 00:27:23.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.22.19.254:44849,127.22.19.253:37879,127.22.19.252:32833 with env {} 00:27:23.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 00:27:23.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:23.392488 22616 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 00:27:23.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:23.392910 22616 flags.cc:406] Enabled unsafe flag: --never_fsync=true 00:27:23.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:23.403678 22616 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 00:27:23.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:23.403833 22616 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 00:27:23.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:23.403940 22616 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 00:27:23.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:23.404058 22616 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true 00:27:23.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.420531 22616 master_main.cc:73] Master server non-default flags: 00:27:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/data 00:27:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/wal 00:27:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 00:27:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.22.19.254:44849,127.22.19.253:37879,127.22.19.252:32833 00:27:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 00:27:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.22.19.254:44849 00:27:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 00:27:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 00:27:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:23.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/data/info.pb 00:27:23.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.22.19.254 00:27:23.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:23.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 00:27:23.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:23.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 00:27:23.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 00:27:23.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:23.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/logs 00:27:23.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 00:27:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 00:27:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-SNAPSHOT 00:27:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5673617937533e44afc5412722108dfed4a7f7b2 00:27:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 00:27:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by None at 20 Feb 2019 00:18:11 UTC on jenkins-slave-8qw8 00:27:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 16477 00:27:23.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 00:27:23.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.436195 22616 master_main.cc:80] Initializing master server... 00:27:23.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.437943 22616 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 83671us 00:27:23.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.438907 22616 fs_manager.cc:263] Metadata directory not provided 00:27:23.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.439054 22616 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 00:27:23.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.439337 22616 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 00:27:23.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.439492 22616 server_base.cc:434] Attempting to create new FS layout instead 00:27:23.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.442723 22616 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/data/instance: 00:27:23.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "3a6002cd74bd48eca090ad5ae522ab7d" 00:27:23.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:23 on dist-test-slave-dist-test-slave-vnz7" 00:27:23.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.443521 22616 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/wal/instance: 00:27:23.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "3a6002cd74bd48eca090ad5ae522ab7d" 00:27:23.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:23 on dist-test-slave-dist-test-slave-vnz7" 00:27:23.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.454396 22616 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.009s sys 0.001s 00:27:23.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.455804 22616 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 00:27:23.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.456334 22616 file_cache.cc:466] Constructed file cache lbm with capacity 419430 00:27:23.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.463450 22616 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 00:27:23.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.463590 22616 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/data,/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/wal 00:27:23.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "3a6002cd74bd48eca090ad5ae522ab7d" 00:27:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:23 on dist-test-slave-dist-test-slave-vnz7" 00:27:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.463855 22616 fs_report.cc:352] FS layout report 00:27:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 00:27:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/wal 00:27:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/wal 00:27:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/data/data 00:27:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 00:27:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 00:27:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 00:27:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 00:27:23.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 00:27:23.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 00:27:23.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 00:27:23.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 00:27:23.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 00:27:23.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 00:27:23.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.929881 22616 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 00:27:23.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.932134 22616 master_main.cc:83] Starting Master server... 00:27:23.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.985991 22616 rpc_server.cc:205] RPC server started. Bound to: 127.22.19.254:44849 00:27:23.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.987252 22616 webserver.cc:175] Starting webserver on 127.22.19.254:0 00:27:23.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.987393 22616 webserver.cc:186] Document root disabled 00:27:23.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.988894 22616 webserver.cc:313] Webserver started. Bound to: http://127.22.19.254:38817/ 00:27:23.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.989665 22616 server_base.cc:608] Dumped server information to /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-0/data/info.pb 00:27:23.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.993189 22677 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:23.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.996940 22607 external_mini_cluster.cc:995] Started /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master as pid 22616 00:27:23.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:23.997902 22607 external_mini_cluster.cc:933] Running /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master 00:27:23.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master 00:27:23.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/wal 00:27:23.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/data 00:27:23.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 00:27:23.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 00:27:23.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 00:27:23.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 00:27:23.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.22.19.253:37879 00:27:23.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.22.19.253 00:27:23.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:23.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 00:27:23.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:23.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:23.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:24.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:24.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 00:27:24.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:24.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/logs 00:27:24.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/data/info.pb 00:27:24.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:24.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 00:27:24.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 00:27:24.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 00:27:24.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 00:27:24.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.22.19.254:44849,127.22.19.253:37879,127.22.19.252:32833 with env {} 00:27:24.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.005246 22677 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } has no permanent_uuid. Determining permanent_uuid... 00:27:24.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.030092 22677 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } has no permanent_uuid. Determining permanent_uuid... 00:27:24.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.031675 22677 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.253:37879: Network error: Client connection negotiation failed: client connection to 127.22.19.253:37879: connect: Connection refused (error 111) 00:27:24.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.051084 22677 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } attempt: 1 00:27:24.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.052665 22677 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.253:37879: Network error: Client connection negotiation failed: client connection to 127.22.19.253:37879: connect: Connection refused (error 111) 00:27:24.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.101017 22677 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } attempt: 2 00:27:24.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.102670 22677 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.253:37879: Network error: Client connection negotiation failed: client connection to 127.22.19.253:37879: connect: Connection refused (error 111) 00:27:24.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 00:27:24.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.142980 22678 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 00:27:24.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.143448 22678 flags.cc:406] Enabled unsafe flag: --never_fsync=true 00:27:24.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.154371 22678 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 00:27:24.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.154536 22678 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 00:27:24.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.154639 22678 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 00:27:24.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.154762 22678 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true 00:27:24.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.171242 22678 master_main.cc:73] Master server non-default flags: 00:27:24.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/data 00:27:24.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/wal 00:27:24.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.22.19.254:44849,127.22.19.253:37879,127.22.19.252:32833 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.22.19.253:37879 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/data/info.pb 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.22.19.253 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 00:27:24.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:24.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/logs 00:27:24.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:24.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 00:27:24.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 00:27:24.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-SNAPSHOT 00:27:24.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5673617937533e44afc5412722108dfed4a7f7b2 00:27:24.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 00:27:24.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by None at 20 Feb 2019 00:18:11 UTC on jenkins-slave-8qw8 00:27:24.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 16477 00:27:24.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 00:27:24.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.184626 22678 master_main.cc:80] Initializing master server... 00:27:24.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.186282 22678 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 84171us 00:27:24.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.187069 22678 fs_manager.cc:263] Metadata directory not provided 00:27:24.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.187211 22678 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 00:27:24.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.187487 22678 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 00:27:24.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.187613 22678 server_base.cc:434] Attempting to create new FS layout instead 00:27:24.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.190578 22678 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/data/instance: 00:27:24.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "efd8e1f777454c9798900ea321457574" 00:27:24.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:24 on dist-test-slave-dist-test-slave-vnz7" 00:27:24.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.191280 22678 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/wal/instance: 00:27:24.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "efd8e1f777454c9798900ea321457574" 00:27:24.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:24 on dist-test-slave-dist-test-slave-vnz7" 00:27:24.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.201732 22678 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s 00:27:24.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.203097 22678 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 00:27:24.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.203624 22678 file_cache.cc:466] Constructed file cache lbm with capacity 419430 00:27:24.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.210592 22678 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 00:27:24.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.210737 22678 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/data,/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/wal 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "efd8e1f777454c9798900ea321457574" 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:24 on dist-test-slave-dist-test-slave-vnz7" 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.210948 22678 fs_report.cc:352] FS layout report 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/wal 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/wal 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/data/data 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 00:27:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 00:27:24.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 00:27:24.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 00:27:24.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 00:27:24.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.213981 22677 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } attempt: 3 00:27:24.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.215474 22677 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.253:37879: Network error: Client connection negotiation failed: client connection to 127.22.19.253:37879: connect: Connection refused (error 111) 00:27:24.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.391768 22677 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } attempt: 4 00:27:24.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.393321 22677 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.253:37879: Network error: Client connection negotiation failed: client connection to 127.22.19.253:37879: connect: Connection refused (error 111) 00:27:24.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.669836 22678 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 00:27:24.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.671991 22678 master_main.cc:83] Starting Master server... 00:27:24.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.696622 22677 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } attempt: 5 00:27:24.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.698393 22677 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.253:37879: Network error: Client connection negotiation failed: client connection to 127.22.19.253:37879: connect: Connection refused (error 111) 00:27:24.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.727026 22678 rpc_server.cc:205] RPC server started. Bound to: 127.22.19.253:37879 00:27:24.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.727955 22678 webserver.cc:175] Starting webserver on 127.22.19.253:0 00:27:24.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.728075 22678 webserver.cc:186] Document root disabled 00:27:24.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.729444 22678 webserver.cc:313] Webserver started. Bound to: http://127.22.19.253:33661/ 00:27:24.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.730183 22678 server_base.cc:608] Dumped server information to /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-1/data/info.pb 00:27:24.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.733224 22741 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:24.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.734975 22607 external_mini_cluster.cc:995] Started /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master as pid 22678 00:27:24.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.736011 22607 external_mini_cluster.cc:933] Running /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master 00:27:24.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master 00:27:24.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/wal 00:27:24.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/data 00:27:24.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 00:27:24.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 00:27:24.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 00:27:24.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 00:27:24.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.22.19.252:32833 00:27:24.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.22.19.252 00:27:24.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:24.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/logs 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/data/info.pb 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 00:27:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.22.19.254:44849,127.22.19.253:37879,127.22.19.252:32833 with env {} 00:27:24.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.745115 22741 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } has no permanent_uuid. Determining permanent_uuid... 00:27:24.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.764559 22741 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } has no permanent_uuid. Determining permanent_uuid... 00:27:24.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.776326 22741 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } has no permanent_uuid. Determining permanent_uuid... 00:27:24.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.777802 22741 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.252:32833: Network error: Client connection negotiation failed: client connection to 127.22.19.252:32833: connect: Connection refused (error 111) 00:27:24.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.797133 22741 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } attempt: 1 00:27:24.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.798349 22741 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.252:32833: Network error: Client connection negotiation failed: client connection to 127.22.19.252:32833: connect: Connection refused (error 111) 00:27:24.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.846693 22741 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } attempt: 2 00:27:24.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.848306 22741 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.252:32833: Network error: Client connection negotiation failed: client connection to 127.22.19.252:32833: connect: Connection refused (error 111) 00:27:24.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 00:27:24.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.875164 22742 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 00:27:24.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.875633 22742 flags.cc:406] Enabled unsafe flag: --never_fsync=true 00:27:24.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.886684 22742 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 00:27:24.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.886855 22742 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 00:27:24.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.886958 22742 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 00:27:24.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.887075 22742 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.903774 22742 master_main.cc:73] Master server non-default flags: 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/data 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/wal 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.22.19.254:44849,127.22.19.253:37879,127.22.19.252:32833 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.22.19.252:32833 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/data/info.pb 00:27:24.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.22.19.252 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/logs 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-SNAPSHOT 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5673617937533e44afc5412722108dfed4a7f7b2 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by None at 20 Feb 2019 00:18:11 UTC on jenkins-slave-8qw8 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 16477 00:27:24.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 00:27:24.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.916900 22742 master_main.cc:80] Initializing master server... 00:27:24.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.918527 22742 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 84171us 00:27:24.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.919425 22742 fs_manager.cc:263] Metadata directory not provided 00:27:24.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.919566 22742 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 00:27:24.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.919834 22742 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 00:27:24.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.919960 22742 server_base.cc:434] Attempting to create new FS layout instead 00:27:24.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.922956 22742 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/data/instance: 00:27:24.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "02a20b80b4f7488e9b6969d470490d29" 00:27:24.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:24 on dist-test-slave-dist-test-slave-vnz7" 00:27:24.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.923686 22742 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/wal/instance: 00:27:24.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "02a20b80b4f7488e9b6969d470490d29" 00:27:24.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:24 on dist-test-slave-dist-test-slave-vnz7" 00:27:24.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.934062 22742 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.009s sys 0.001s 00:27:24.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.935438 22742 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 00:27:24.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.935907 22742 file_cache.cc:466] Constructed file cache lbm with capacity 419430 00:27:24.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.942838 22742 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.942970 22742 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/data,/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/wal 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "02a20b80b4f7488e9b6969d470490d29" 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:24 on dist-test-slave-dist-test-slave-vnz7" 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.943198 22742 fs_report.cc:352] FS layout report 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/wal 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/wal 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/data/data 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 00:27:24.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 00:27:24.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 00:27:24.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 00:27:24.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 00:27:24.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 00:27:24.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 00:27:24.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:24.959604 22741 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } attempt: 3 00:27:24.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:24.961191 22741 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.252:32833: Network error: Client connection negotiation failed: client connection to 127.22.19.252:32833: connect: Connection refused (error 111) 00:27:25.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.137507 22741 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } attempt: 4 00:27:25.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.139034 22741 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.252:32833: Network error: Client connection negotiation failed: client connection to 127.22.19.252:32833: connect: Connection refused (error 111) 00:27:25.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.232676 22677 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } attempt: 6 00:27:25.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.242708 22677 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } has no permanent_uuid. Determining permanent_uuid... 00:27:25.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.244113 22677 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.252:32833: Network error: Client connection negotiation failed: client connection to 127.22.19.252:32833: connect: Connection refused (error 111) 00:27:25.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.276500 22677 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } attempt: 1 00:27:25.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.278211 22677 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.252:32833: Network error: Client connection negotiation failed: client connection to 127.22.19.252:32833: connect: Connection refused (error 111) 00:27:25.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.346531 22677 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } attempt: 2 00:27:25.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.348179 22677 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.252:32833: Network error: Client connection negotiation failed: client connection to 127.22.19.252:32833: connect: Connection refused (error 111) 00:27:25.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.402901 22742 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 00:27:25.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.405149 22742 master_main.cc:83] Starting Master server... 00:27:25.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.424445 22677 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } attempt: 3 00:27:25.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.426112 22677 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.252:32833: Network error: Client connection negotiation failed: client connection to 127.22.19.252:32833: connect: Connection refused (error 111) 00:27:25.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.442303 22741 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } attempt: 5 00:27:25.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.443879 22741 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.22.19.252:32833: Network error: Client connection negotiation failed: client connection to 127.22.19.252:32833: connect: Connection refused (error 111) 00:27:25.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.463770 22742 rpc_server.cc:205] RPC server started. Bound to: 127.22.19.252:32833 00:27:25.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.464732 22742 webserver.cc:175] Starting webserver on 127.22.19.252:0 00:27:25.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.464843 22742 webserver.cc:186] Document root disabled 00:27:25.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.466236 22742 webserver.cc:313] Webserver started. Bound to: http://127.22.19.252:42511/ 00:27:25.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.467097 22742 server_base.cc:608] Dumped server information to /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/master-2/data/info.pb 00:27:25.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.470036 22807 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:25.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.472821 22607 external_mini_cluster.cc:995] Started /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master as pid 22742 00:27:25.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.473834 22607 external_mini_cluster.cc:933] Running /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver 00:27:25.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver 00:27:25.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/wal 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/data 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.22.19.193:0 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.22.19.193 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.22.19.193 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.22.19.254:44849,127.22.19.253:37879,127.22.19.252:32833 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:25.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/logs 00:27:25.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/data/info.pb 00:27:25.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:25.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 00:27:25.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 00:27:25.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags with env {} 00:27:25.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.481559 22807 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } has no permanent_uuid. Determining permanent_uuid... 00:27:25.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.499971 22807 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } has no permanent_uuid. Determining permanent_uuid... 00:27:25.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.508348 22807 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } has no permanent_uuid. Determining permanent_uuid... 00:27:25.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.525494 22807 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29: Bootstrap starting. 00:27:25.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.527992 22807 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29: No blocks or log segments found. Creating new log. 00:27:25.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.529256 22807 log.cc:525] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29: Log is configured to *not* fsync() on all Append() calls 00:27:25.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.532874 22807 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29: No bootstrap required, opened a new log 00:27:25.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.544243 22807 raft_consensus.cc:340] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } 00:27:25.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.544795 22807 raft_consensus.cc:366] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:25.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.544948 22807 raft_consensus.cc:705] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a20b80b4f7488e9b6969d470490d29, State: Initialized, Role: FOLLOWER 00:27:25.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.545451 22807 consensus_queue.cc:229] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } 00:27:25.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.547864 22807 sys_catalog.cc:337] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:25.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.548296 22807 sys_catalog.cc:340] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [sys.catalog]: This master's current role is: FOLLOWER 00:27:25.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.549101 22807 sys_catalog.cc:424] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [sys.catalog]: configured and running, proceeding with master startup. 00:27:25.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.563432 22677 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } attempt: 4 00:27:25.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.579154 22677 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d: Bootstrap starting. 00:27:25.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.582201 22677 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d: No blocks or log segments found. Creating new log. 00:27:25.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.583803 22677 log.cc:525] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d: Log is configured to *not* fsync() on all Append() calls 00:27:25.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.589440 22677 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d: No bootstrap required, opened a new log 00:27:25.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 00:27:25.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.596139 22808 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 00:27:25.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.596860 22808 flags.cc:406] Enabled unsafe flag: --never_fsync=true 00:27:25.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.602313 22677 raft_consensus.cc:340] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } 00:27:25.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.603097 22677 raft_consensus.cc:366] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:25.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.603420 22677 raft_consensus.cc:705] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6002cd74bd48eca090ad5ae522ab7d, State: Initialized, Role: FOLLOWER 00:27:25.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.604210 22677 consensus_queue.cc:229] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } 00:27:25.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.606274 22808 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 00:27:25.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.606132 22677 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:25.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.606792 22808 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.19.193 00:27:25.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.606912 22677 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [sys.catalog]: This master's current role is: FOLLOWER 00:27:25.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.608634 22677 sys_catalog.cc:424] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [sys.catalog]: configured and running, proceeding with master startup. 00:27:25.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.548210 22813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:25.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.617614 22813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [sys.catalog]: This master's current role is: FOLLOWER 00:27:25.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.620249 22742 master_main.cc:86] Master server successfully started. 00:27:25.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.623659 22808 tablet_server_main.cc:78] Tablet server non-default flags: 00:27:25.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/data 00:27:25.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/wal 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.22.19.193:0 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/data/info.pb 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.22.19.193 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.22.19.254:44849,127.22.19.253:37879,127.22.19.252:32833 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.22.19.193 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/logs 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-SNAPSHOT 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5673617937533e44afc5412722108dfed4a7f7b2 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by None at 20 Feb 2019 00:18:11 UTC on jenkins-slave-8qw8 00:27:25.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 16477 00:27:25.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 00:27:25.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.636947 22808 tablet_server_main.cc:85] Initializing tablet server... 00:27:25.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.636961 22814 catalog_manager.cc:1104] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29: acquiring CA information for follower catalog manager: Not found: root CA entry not found 00:27:25.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.637140 22814 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 00:27:25.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.648828 22808 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 84671us 00:27:25.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.649524 22808 fs_manager.cc:263] Metadata directory not provided 00:27:25.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.650244 22808 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 00:27:25.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.650642 22808 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 00:27:25.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.650861 22808 server_base.cc:434] Attempting to create new FS layout instead 00:27:25.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.653898 22808 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/data/instance: 00:27:25.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "5ab94ed628b14cf09ec9bd11be159caa" 00:27:25.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:25 on dist-test-slave-dist-test-slave-vnz7" 00:27:25.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.654798 22808 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/wal/instance: 00:27:25.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "5ab94ed628b14cf09ec9bd11be159caa" 00:27:25.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:25 on dist-test-slave-dist-test-slave-vnz7" 00:27:25.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.657213 22763 tablet.cc:1786] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29: Can't schedule compaction. Clean time has not been advanced past its initial value. 00:27:25.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.668841 22808 fs_manager.cc:503] Time spent creating directory manager: real 0.013s user 0.007s sys 0.005s 00:27:25.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.670568 22808 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 00:27:25.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.671077 22808 file_cache.cc:466] Constructed file cache lbm with capacity 419430 00:27:25.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.606473 22816 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:25.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.686017 22816 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [sys.catalog]: This master's current role is: FOLLOWER 00:27:25.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.688625 22808 fs_manager.cc:419] Time spent opening block manager: real 0.008s user 0.000s sys 0.005s 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.688781 22808 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/data,/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/wal 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "5ab94ed628b14cf09ec9bd11be159caa" 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:25 on dist-test-slave-dist-test-slave-vnz7" 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.689035 22808 fs_report.cc:352] FS layout report 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/wal 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/wal 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/data/data 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 00:27:25.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 00:27:25.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.685070 22633 tablet.cc:1786] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d: Can't schedule compaction. Clean time has not been advanced past its initial value. 00:27:25.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.712472 22616 master_main.cc:86] Master server successfully started. 00:27:25.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.716922 22817 catalog_manager.cc:1104] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d: acquiring CA information for follower catalog manager: Not found: root CA entry not found 00:27:25.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:25.717425 22817 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 00:27:25.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.897541 22813 raft_consensus.cc:472] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 00:27:25.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.897945 22813 raft_consensus.cc:494] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } 00:27:25.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.899926 22813 leader_election.cc:296] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a6002cd74bd48eca090ad5ae522ab7d (127.22.19.254:44849), efd8e1f777454c9798900ea321457574 (127.22.19.253:37879) 00:27:25.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.901441 22654 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02a20b80b4f7488e9b6969d470490d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" is_pre_election: true 00:27:25.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.902019 22654 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a20b80b4f7488e9b6969d470490d29 in term 0. 00:27:25.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.903417 22753 leader_election.cc:310] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a20b80b4f7488e9b6969d470490d29, 3a6002cd74bd48eca090ad5ae522ab7d; no voters: 00:27:25.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.904170 22813 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [term 0 FOLLOWER]: Leader pre-election won for term 1 00:27:25.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.904336 22813 raft_consensus.cc:472] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 00:27:25.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.904503 22813 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [term 0 FOLLOWER]: Advancing to term 1 00:27:25.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.908767 22813 raft_consensus.cc:494] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } 00:27:25.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.910800 22654 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02a20b80b4f7488e9b6969d470490d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" 00:27:25.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.911108 22654 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [term 0 FOLLOWER]: Advancing to term 1 00:27:25.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.914436 22654 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a20b80b4f7488e9b6969d470490d29 in term 1. 00:27:25.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.919654 22753 leader_election.cc:310] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a20b80b4f7488e9b6969d470490d29, 3a6002cd74bd48eca090ad5ae522ab7d; no voters: 00:27:25.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.923995 22831 raft_consensus.cc:2640] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [term 1 FOLLOWER]: Leader election won for term 1 00:27:25.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.939527 22813 leader_election.cc:296] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [CANDIDATE]: Term 1 election: Requested vote from peers 3a6002cd74bd48eca090ad5ae522ab7d (127.22.19.254:44849), efd8e1f777454c9798900ea321457574 (127.22.19.253:37879) 00:27:25.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.952108 22831 raft_consensus.cc:667] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [term 1 LEADER]: Becoming Leader. State: Replica: 02a20b80b4f7488e9b6969d470490d29, State: Running, Role: LEADER 00:27:25.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.952575 22831 consensus_queue.cc:206] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } 00:27:25.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.957191 22813 sys_catalog.cc:337] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02a20b80b4f7488e9b6969d470490d29. Latest consensus state: current_term: 1 leader_uuid: "02a20b80b4f7488e9b6969d470490d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:25.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.958400 22813 sys_catalog.cc:340] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [sys.catalog]: This master's current role is: LEADER 00:27:25.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.964468 22834 catalog_manager.cc:1046] Loading table and tablet metadata into memory... 00:27:25.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.974757 22834 catalog_manager.cc:1057] Initializing Kudu internal certificate authority... 00:27:25.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.978137 22741 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } attempt: 6 00:27:25.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.994431 22741 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574: Bootstrap starting. 00:27:25.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.997292 22741 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574: No blocks or log segments found. Creating new log. 00:27:25.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:25.999651 22741 log.cc:525] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574: Log is configured to *not* fsync() on all Append() calls 00:27:26.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.010646 22741 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574: No bootstrap required, opened a new log 00:27:26.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.014736 22654 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [term 1 FOLLOWER]: Refusing update from remote peer 02a20b80b4f7488e9b6969d470490d29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 00:27:26.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.017000 22813 consensus_queue.cc:984] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 00:27:26.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.032459 22816 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02a20b80b4f7488e9b6969d470490d29. Latest consensus state: current_term: 1 leader_uuid: "02a20b80b4f7488e9b6969d470490d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:26.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.032929 22816 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [sys.catalog]: This master's current role is: FOLLOWER 00:27:26.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.040465 22831 sys_catalog.cc:337] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02a20b80b4f7488e9b6969d470490d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:26.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.041023 22831 sys_catalog.cc:340] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [sys.catalog]: This master's current role is: LEADER 00:27:26.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.042769 22837 mvcc.cc:203] Tried to move safe_time back from 6351349538855931904 to 6351349538627006464. Current Snapshot: MvccSnapshot[committed={T|T < 6351349538855931904}] 00:27:26.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.044896 22834 catalog_manager.cc:941] Generated new certificate authority record 00:27:26.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.046303 22834 catalog_manager.cc:1066] Loading token signing keys... 00:27:26.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.051635 22838 mvcc.cc:203] Tried to move safe_time back from 6351349538855931904 to 6351349538627006464. Current Snapshot: MvccSnapshot[committed={T|T < 6351349538855931904}] 00:27:26.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.057960 22816 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02a20b80b4f7488e9b6969d470490d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:26.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.058445 22816 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d [sys.catalog]: This master's current role is: FOLLOWER 00:27:26.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.061563 22741 raft_consensus.cc:340] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } 00:27:26.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.066342 22741 raft_consensus.cc:366] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:26.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.066658 22741 raft_consensus.cc:705] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd8e1f777454c9798900ea321457574, State: Initialized, Role: FOLLOWER 00:27:26.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.067392 22741 consensus_queue.cc:229] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } 00:27:26.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.070606 22842 sys_catalog.cc:337] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:26.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.071121 22842 sys_catalog.cc:340] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [sys.catalog]: This master's current role is: FOLLOWER 00:27:26.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.075583 22741 sys_catalog.cc:337] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:26.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.076197 22741 sys_catalog.cc:340] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [sys.catalog]: This master's current role is: FOLLOWER 00:27:26.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.077241 22741 sys_catalog.cc:424] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [sys.catalog]: configured and running, proceeding with master startup. 00:27:26.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.128831 22834 catalog_manager.cc:4095] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29: Generated new TSK 0 00:27:26.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.077353 22718 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02a20b80b4f7488e9b6969d470490d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd8e1f777454c9798900ea321457574" is_pre_election: true 00:27:26.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.216737 22718 raft_consensus.cc:2346] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a20b80b4f7488e9b6969d470490d29 in term 0. 00:27:26.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.217412 22717 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02a20b80b4f7488e9b6969d470490d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efd8e1f777454c9798900ea321457574" 00:27:26.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.217767 22717 raft_consensus.cc:2886] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [term 0 FOLLOWER]: Advancing to term 1 00:27:26.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.225337 22717 raft_consensus.cc:2346] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a20b80b4f7488e9b6969d470490d29 in term 1. 00:27:26.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.226300 22716 raft_consensus.cc:1181] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [term 1 FOLLOWER]: Refusing update from remote peer 02a20b80b4f7488e9b6969d470490d29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 00:27:26.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.228523 22831 consensus_queue.cc:984] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 00:27:26.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:26.174923 22697 tablet.cc:1786] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574: Can't schedule compaction. Clean time has not been advanced past its initial value. 00:27:26.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.266553 22678 master_main.cc:86] Master server successfully started. 00:27:26.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.271085 22808 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 00:27:26.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.273399 22845 mvcc.cc:203] Tried to move safe_time back from 6351349538855931904 to 6351349538627006464. Current Snapshot: MvccSnapshot[committed={T|T < 6351349538855931904}] 00:27:26.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.276726 22808 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) 00:27:26.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.283414 22808 tablet_server_main.cc:90] Starting tablet server... 00:27:26.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.283640 22842 sys_catalog.cc:337] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02a20b80b4f7488e9b6969d470490d29. Latest consensus state: current_term: 1 leader_uuid: "02a20b80b4f7488e9b6969d470490d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:26.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.284216 22842 sys_catalog.cc:340] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [sys.catalog]: This master's current role is: FOLLOWER 00:27:26.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.283816 22850 sys_catalog.cc:337] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02a20b80b4f7488e9b6969d470490d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:26.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.284780 22850 sys_catalog.cc:340] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574 [sys.catalog]: This master's current role is: FOLLOWER 00:27:26.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.291326 22843 catalog_manager.cc:1102] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574: acquiring CA information for follower catalog manager: success 00:27:26.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.293462 22843 catalog_manager.cc:1130] T 00000000000000000000000000000000 P efd8e1f777454c9798900ea321457574: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 00:27:26.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.724043 22817 catalog_manager.cc:1102] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d: acquiring CA information for follower catalog manager: success 00:27:26.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.727327 22817 catalog_manager.cc:1130] T 00000000000000000000000000000000 P 3a6002cd74bd48eca090ad5ae522ab7d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 00:27:26.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.772111 22831 sys_catalog.cc:337] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02a20b80b4f7488e9b6969d470490d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6002cd74bd48eca090ad5ae522ab7d" member_type: VOTER last_known_addr { host: "127.22.19.254" port: 44849 } } peers { permanent_uuid: "efd8e1f777454c9798900ea321457574" member_type: VOTER last_known_addr { host: "127.22.19.253" port: 37879 } } peers { permanent_uuid: "02a20b80b4f7488e9b6969d470490d29" member_type: VOTER last_known_addr { host: "127.22.19.252" port: 32833 } } } 00:27:26.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.773258 22831 sys_catalog.cc:340] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 [sys.catalog]: This master's current role is: LEADER 00:27:26.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.903920 22808 rpc_server.cc:205] RPC server started. Bound to: 127.22.19.193:35627 00:27:26.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.905524 22808 webserver.cc:175] Starting webserver on 127.22.19.193:0 00:27:26.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.905688 22808 webserver.cc:186] Document root disabled 00:27:26.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.908767 22808 webserver.cc:313] Webserver started. Bound to: http://127.22.19.193:44667/ 00:27:26.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.909821 22808 server_base.cc:608] Dumped server information to /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-0/data/info.pb 00:27:26.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.917201 22607 external_mini_cluster.cc:995] Started /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver as pid 22808 00:27:26.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.918443 22607 external_mini_cluster.cc:933] Running /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver 00:27:26.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver 00:27:26.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/wal 00:27:26.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/data 00:27:26.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 00:27:26.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.22.19.194:0 00:27:26.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.22.19.194 00:27:26.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.22.19.194 00:27:26.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.22.19.254:44849,127.22.19.253:37879,127.22.19.252:32833 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/logs 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/data/info.pb 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 00:27:26.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags with env {} 00:27:26.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:26.968135 22808 tablet_server_main.cc:93] Tablet server successfully started. 00:27:27.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.018959 22964 heartbeater.cc:345] Connected to a master server at 127.22.19.253:37879 00:27:27.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.019366 22964 heartbeater.cc:426] Registering TS with master... 00:27:27.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.020064 22964 heartbeater.cc:475] Master 127.22.19.253:37879 requested a full tablet report, sending... 00:27:27.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.024428 22708 ts_manager.cc:98] Registered new tserver with Master: 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627) 00:27:27.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.034449 22970 heartbeater.cc:345] Connected to a master server at 127.22.19.254:44849 00:27:27.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.034727 22970 heartbeater.cc:426] Registering TS with master... 00:27:27.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.035262 22970 heartbeater.cc:475] Master 127.22.19.254:44849 requested a full tablet report, sending... 00:27:27.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.039315 22644 ts_manager.cc:98] Registered new tserver with Master: 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627) 00:27:27.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.042270 22963 heartbeater.cc:345] Connected to a master server at 127.22.19.252:32833 00:27:27.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.042529 22963 heartbeater.cc:426] Registering TS with master... 00:27:27.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.042968 22963 heartbeater.cc:475] Master 127.22.19.252:32833 requested a full tablet report, sending... 00:27:27.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.045053 22774 ts_manager.cc:98] Registered new tserver with Master: 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627) 00:27:27.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.047942 22774 master_service.cc:269] Signed X509 certificate for tserver {username='slave'} at 127.22.19.193:47609 00:27:27.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 00:27:27.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:27.255807 22969 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 00:27:27.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:27.256773 22969 flags.cc:406] Enabled unsafe flag: --never_fsync=true 00:27:27.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:27.289326 22969 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 00:27:27.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:27.290206 22969 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.19.194 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.323772 22969 tablet_server_main.cc:78] Tablet server non-default flags: 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/data 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/wal 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.22.19.194:0 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/data/info.pb 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.22.19.194 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.22.19.254:44849,127.22.19.253:37879,127.22.19.252:32833 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 00:27:27.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.22.19.194 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/logs 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-SNAPSHOT 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5673617937533e44afc5412722108dfed4a7f7b2 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by None at 20 Feb 2019 00:18:11 UTC on jenkins-slave-8qw8 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 16477 00:27:27.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 00:27:27.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.351225 22969 tablet_server_main.cc:85] Initializing tablet server... 00:27:27.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.355208 22969 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 85671us 00:27:27.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.361037 22969 fs_manager.cc:263] Metadata directory not provided 00:27:27.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.361528 22969 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 00:27:27.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.362072 22969 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 00:27:27.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.362385 22969 server_base.cc:434] Attempting to create new FS layout instead 00:27:27.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.366848 22969 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/data/instance: 00:27:27.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1313c80b0ed640658101132ab87b8fc5" 00:27:27.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:27 on dist-test-slave-dist-test-slave-vnz7" 00:27:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.371091 22969 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/wal/instance: 00:27:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1313c80b0ed640658101132ab87b8fc5" 00:27:27.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:27 on dist-test-slave-dist-test-slave-vnz7" 00:27:27.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.393517 22969 fs_manager.cc:503] Time spent creating directory manager: real 0.022s user 0.016s sys 0.004s 00:27:27.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.395526 22969 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 00:27:27.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.396423 22969 file_cache.cc:466] Constructed file cache lbm with capacity 419430 00:27:27.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.448540 22969 fs_manager.cc:419] Time spent opening block manager: real 0.015s user 0.001s sys 0.000s 00:27:27.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.449105 22969 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/data,/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/wal 00:27:27.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1313c80b0ed640658101132ab87b8fc5" 00:27:27.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:27 on dist-test-slave-dist-test-slave-vnz7" 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:27.449743 22969 fs_report.cc:352] FS layout report 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/wal 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/wal 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/data/data 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 00:27:27.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 00:27:28.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.051079 22963 heartbeater.cc:467] Master 127.22.19.252:32833 was elected leader, sending a full tablet report... 00:27:28.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:28.148025 22969 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.663s user 0.124s sys 0.385s 00:27:28.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:28.148232 22969 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.663s user 0.124s sys 0.385s 00:27:28.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.173751 22969 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 00:27:28.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.181633 22969 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) 00:27:28.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.183913 22969 tablet_server_main.cc:90] Starting tablet server... 00:27:28.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.379614 22969 rpc_server.cc:205] RPC server started. Bound to: 127.22.19.194:43751 00:27:28.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.380517 22969 webserver.cc:175] Starting webserver on 127.22.19.194:0 00:27:28.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.380625 22969 webserver.cc:186] Document root disabled 00:27:28.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.382758 22969 webserver.cc:313] Webserver started. Bound to: http://127.22.19.194:40195/ 00:27:28.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.383474 22969 server_base.cc:608] Dumped server information to /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-1/data/info.pb 00:27:28.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.390789 22607 external_mini_cluster.cc:995] Started /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver as pid 22969 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.391949 22607 external_mini_cluster.cc:933] Running /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/wal 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/data 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.22.19.195:0 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.22.19.195 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.22.19.195 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.22.19.254:44849,127.22.19.253:37879,127.22.19.252:32833 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 00:27:28.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:28.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/logs 00:27:28.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/data/info.pb 00:27:28.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:28.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 00:27:28.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 00:27:28.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags with env {} 00:27:28.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.426172 22969 tablet_server_main.cc:93] Tablet server successfully started. 00:27:28.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.467216 23095 heartbeater.cc:345] Connected to a master server at 127.22.19.253:37879 00:27:28.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.467614 23095 heartbeater.cc:426] Registering TS with master... 00:27:28.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.468854 23095 heartbeater.cc:475] Master 127.22.19.253:37879 requested a full tablet report, sending... 00:27:28.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.475309 22708 ts_manager.cc:98] Registered new tserver with Master: 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194:43751) 00:27:28.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.468691 23094 heartbeater.cc:345] Connected to a master server at 127.22.19.252:32833 00:27:28.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.480718 23094 heartbeater.cc:426] Registering TS with master... 00:27:28.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.481304 23094 heartbeater.cc:475] Master 127.22.19.252:32833 requested a full tablet report, sending... 00:27:28.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.471637 23096 heartbeater.cc:345] Connected to a master server at 127.22.19.254:44849 00:27:28.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.482239 23096 heartbeater.cc:426] Registering TS with master... 00:27:28.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.484947 22774 ts_manager.cc:98] Registered new tserver with Master: 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194:43751) 00:27:28.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.486899 22774 master_service.cc:269] Signed X509 certificate for tserver {username='slave'} at 127.22.19.194:37147 00:27:28.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.482723 23096 heartbeater.cc:475] Master 127.22.19.254:44849 requested a full tablet report, sending... 00:27:28.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.489820 22644 ts_manager.cc:98] Registered new tserver with Master: 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194:43751) 00:27:28.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 00:27:28.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:28.560335 23098 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 00:27:28.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:28.560737 23098 flags.cc:406] Enabled unsafe flag: --never_fsync=true 00:27:28.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:28.571044 23098 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 00:27:28.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:28.571426 23098 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.19.195 00:27:28.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.587553 23098 tablet_server_main.cc:78] Tablet server non-default flags: 00:27:28.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/data 00:27:28.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/wal 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.22.19.195:0 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/data/info.pb 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.22.19.195 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.22.19.254:44849,127.22.19.253:37879,127.22.19.252:32833 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.22.19.195 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/logs 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-SNAPSHOT 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5673617937533e44afc5412722108dfed4a7f7b2 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by None at 20 Feb 2019 00:18:11 UTC on jenkins-slave-8qw8 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 16477 00:27:28.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 00:27:28.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.602865 23098 tablet_server_main.cc:85] Initializing tablet server... 00:27:28.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.604704 23098 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 86171us 00:27:28.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.605136 23098 fs_manager.cc:263] Metadata directory not provided 00:27:28.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.605259 23098 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 00:27:28.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.605543 23098 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 00:27:28.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.605659 23098 server_base.cc:434] Attempting to create new FS layout instead 00:27:28.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.608633 23098 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/data/instance: 00:27:28.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "c9ad861f849642ad801dde9f5e92c23f" 00:27:28.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:28 on dist-test-slave-dist-test-slave-vnz7" 00:27:28.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.609334 23098 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/wal/instance: 00:27:28.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "c9ad861f849642ad801dde9f5e92c23f" 00:27:28.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:28 on dist-test-slave-dist-test-slave-vnz7" 00:27:28.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.620874 23098 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.010s sys 0.001s 00:27:28.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.622104 23098 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 00:27:28.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.622588 23098 file_cache.cc:466] Constructed file cache lbm with capacity 419430 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.629022 23098 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.629159 23098 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/data,/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/wal 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "c9ad861f849642ad801dde9f5e92c23f" 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:28 on dist-test-slave-dist-test-slave-vnz7" 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:28.629384 23098 fs_report.cc:352] FS layout report 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/wal 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/wal 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/data/data 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 00:27:28.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 00:27:28.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 00:27:28.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 00:27:28.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 00:27:29.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.132274 23098 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 00:27:29.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.137557 23098 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) 00:27:29.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.138921 23098 tablet_server_main.cc:90] Starting tablet server... 00:27:29.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.290922 23098 rpc_server.cc:205] RPC server started. Bound to: 127.22.19.195:44599 00:27:29.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.291934 23098 webserver.cc:175] Starting webserver on 127.22.19.195:0 00:27:29.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.292045 23098 webserver.cc:186] Document root disabled 00:27:29.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.293882 23098 webserver.cc:313] Webserver started. Bound to: http://127.22.19.195:44869/ 00:27:29.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.294555 23098 server_base.cc:608] Dumped server information to /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster4061125272195581684/ts-2/data/info.pb 00:27:29.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.298897 22607 external_mini_cluster.cc:995] Started /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver as pid 23098 00:27:29.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.325151 23098 tablet_server_main.cc:93] Tablet server successfully started. 00:27:29.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.359486 23228 heartbeater.cc:345] Connected to a master server at 127.22.19.254:44849 00:27:29.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.359822 23228 heartbeater.cc:426] Registering TS with master... 00:27:29.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.360494 23228 heartbeater.cc:475] Master 127.22.19.254:44849 requested a full tablet report, sending... 00:27:29.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.362574 22644 ts_manager.cc:98] Registered new tserver with Master: c9ad861f849642ad801dde9f5e92c23f (127.22.19.195:44599) 00:27:29.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.359885 23226 heartbeater.cc:345] Connected to a master server at 127.22.19.253:37879 00:27:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.370899 23226 heartbeater.cc:426] Registering TS with master... 00:27:29.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.371337 23226 heartbeater.cc:475] Master 127.22.19.253:37879 requested a full tablet report, sending... 00:27:29.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.361160 23225 heartbeater.cc:345] Connected to a master server at 127.22.19.252:32833 00:27:29.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.372617 23225 heartbeater.cc:426] Registering TS with master... 00:27:29.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.373065 23225 heartbeater.cc:475] Master 127.22.19.252:32833 requested a full tablet report, sending... 00:27:29.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.373790 22708 ts_manager.cc:98] Registered new tserver with Master: c9ad861f849642ad801dde9f5e92c23f (127.22.19.195:44599) 00:27:29.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.374881 22774 ts_manager.cc:98] Registered new tserver with Master: c9ad861f849642ad801dde9f5e92c23f (127.22.19.195:44599) 00:27:29.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.376699 22774 master_service.cc:269] Signed X509 certificate for tserver {username='slave'} at 127.22.19.195:55151 00:27:29.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.381621 22607 external_mini_cluster.cc:623] 3 TS(s) registered with all masters 00:27:29.381 [DEBUG - main] (MiniKuduCluster.java:165) Response: 00:27:29.383 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_masters { } 00:27:29.390 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.22.19.254" port: 44849 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.22.19.253" port: 37879 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.22.19.252" port: 32833 } } } 00:27:29.394 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_tservers { } 00:27:29.396 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_tservers { tservers { id { type: TSERVER index: 0 } bound_rpc_address { host: "127.22.19.193" port: 35627 } } tservers { id { type: TSERVER index: 1 } bound_rpc_address { host: "127.22.19.194" port: 43751 } } tservers { id { type: TSERVER index: 2 } bound_rpc_address { host: "127.22.19.195" port: 44599 } } } 00:27:29.396 [INFO - main] (KuduTestHarness.java:142) Creating a new Kudu client... 00:27:29.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.491623 23094 heartbeater.cc:467] Master 127.22.19.252:32833 was elected leader, sending a full tablet report... 00:27:29.521 [DEBUG - main] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:29.625 [DEBUG - Time-limited test] (NetUtil.java:106) Resolved IP of `127.22.19.252' to /127.22.19.252 in 615968ns 00:27:29.746 [DEBUG - New I/O worker #2] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:29.746 [DEBUG - New I/O worker #1] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:29.746 [DEBUG - New I/O worker #3] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:29.882 [DEBUG - New I/O worker #2] (Negotiator.java:764) Authenticated connection [id: 0x1f20b83e, /127.0.0.1:52556 => /127.22.19.254:44849] using SASL/PLAIN 00:27:29.882 [DEBUG - New I/O worker #1] (Negotiator.java:764) Authenticated connection [id: 0xf7711569, /127.0.0.1:37210 => /127.22.19.252:32833] using SASL/PLAIN 00:27:29.883 [DEBUG - New I/O worker #3] (Negotiator.java:764) Authenticated connection [id: 0x19db2f66, /127.0.0.1:39186 => /127.22.19.253:37879] using SASL/PLAIN 00:27:29.914 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:29.915 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=304), Traces: [0ms] refreshing cache from master, [37ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [52ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [65ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [274ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [274ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [274ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:29.923 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=retry RPC@1726820068 returned Deferred@195270795(state=PENDING, result=null, callback=(continuation of Deferred@1033303349 after retry RPC@1726820068), errback=(continuation of Deferred@1033303349 after retry RPC@1726820068)), so the following Deferred is getting paused: Deferred@1033303349(state=PAUSED, result=Deferred@195270795, callback=org.apache.kudu.client.AsyncKuduClient$1@42d31104 -> org.apache.kudu.client.AsyncKuduClient$2@4d468db4 -> wakeup thread Time-limited test, errback=passthrough -> passthrough -> wakeup thread Time-limited test) 00:27:29.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:29.923517 22774 catalog_manager.cc:1386] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37210: 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "testManyClients" 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 00:27:29.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) split_rows_range_bounds { 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) rows: "\004\001\000\n\000\000\000\004\001\000\024\000\000\000\004\001\000\036\000\000\000" 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:29.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:29.938002 22774 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table testManyClients in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 00:27:29.976 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$1@42d31104@1121128708 returned Deferred@1521886039(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduClient$6@50035a69 -> (continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$1@42d31104@1121128708), errback=passthrough -> (continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$1@42d31104@1121128708)), so the following Deferred is getting paused: Deferred@1033303349(state=PAUSED, result=Deferred@1521886039, callback=org.apache.kudu.client.AsyncKuduClient$2@4d468db4 -> wakeup thread Time-limited test, errback=passthrough -> wakeup thread Time-limited test) 00:27:29.990 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:29.991 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428 returned Deferred@404161571(state=PENDING, result=null, callback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428), errback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428)), so the following Deferred is getting paused: Deferred@1033303349(state=PAUSED, result=Deferred@404161571, callback=wakeup thread Time-limited test, errback=wakeup thread Time-limited test) 00:27:30.003 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906 returned Deferred@404161571(state=PENDING, result=null, callback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906), errback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906)), so the following Deferred is getting paused: Deferred@1229342093(state=PAUSED, result=Deferred@404161571, callback=, errback=) 00:27:30.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.012223 23031 tablet_service.cc:870] Processing CreateTablet for tablet e3df67dd290b4bf6b4b9bc367348c886 (table=testManyClients [id=f846a8dbe51f43e08571b061f393845f]), partition=RANGE (key) PARTITION VALUES < 10 00:27:30.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.013371 23031 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet e3df67dd290b4bf6b4b9bc367348c886. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:30.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.015663 22896 tablet_service.cc:870] Processing CreateTablet for tablet e3df67dd290b4bf6b4b9bc367348c886 (table=testManyClients [id=f846a8dbe51f43e08571b061f393845f]), partition=RANGE (key) PARTITION VALUES < 10 00:27:30.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.015914 23029 tablet_service.cc:870] Processing CreateTablet for tablet d5cb2974aabf40dea3f204d4d5929375 (table=testManyClients [id=f846a8dbe51f43e08571b061f393845f]), partition=RANGE (key) PARTITION 20 <= VALUES < 30 00:27:30.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.016628 23029 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet d5cb2974aabf40dea3f204d4d5929375. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:30.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.016685 22896 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet e3df67dd290b4bf6b4b9bc367348c886. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:30.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.016381 23160 tablet_service.cc:870] Processing CreateTablet for tablet d5cb2974aabf40dea3f204d4d5929375 (table=testManyClients [id=f846a8dbe51f43e08571b061f393845f]), partition=RANGE (key) PARTITION 20 <= VALUES < 30 00:27:30.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.017552 23160 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet d5cb2974aabf40dea3f204d4d5929375. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:30.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.017449 23030 tablet_service.cc:870] Processing CreateTablet for tablet fa4fe6266d3b4911aaac3bcebe15999b (table=testManyClients [id=f846a8dbe51f43e08571b061f393845f]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 00:27:30.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.018159 23030 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fa4fe6266d3b4911aaac3bcebe15999b. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:30.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.020710 23028 tablet_service.cc:870] Processing CreateTablet for tablet 762f4bc1622449b8868b95d67f43e131 (table=testManyClients [id=f846a8dbe51f43e08571b061f393845f]), partition=RANGE (key) PARTITION VALUES >= 30 00:27:30.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.016793 23161 tablet_service.cc:870] Processing CreateTablet for tablet fa4fe6266d3b4911aaac3bcebe15999b (table=testManyClients [id=f846a8dbe51f43e08571b061f393845f]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 00:27:30.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.021368 23028 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 762f4bc1622449b8868b95d67f43e131. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:30.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.021842 23161 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fa4fe6266d3b4911aaac3bcebe15999b. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:30.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.021173 23159 tablet_service.cc:870] Processing CreateTablet for tablet 762f4bc1622449b8868b95d67f43e131 (table=testManyClients [id=f846a8dbe51f43e08571b061f393845f]), partition=RANGE (key) PARTITION VALUES >= 30 00:27:30.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.022879 23159 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 762f4bc1622449b8868b95d67f43e131. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:30.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.023476 23162 tablet_service.cc:870] Processing CreateTablet for tablet e3df67dd290b4bf6b4b9bc367348c886 (table=testManyClients [id=f846a8dbe51f43e08571b061f393845f]), partition=RANGE (key) PARTITION VALUES < 10 00:27:30.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.024641 23162 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet e3df67dd290b4bf6b4b9bc367348c886. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:30.042 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773 returned Deferred@404161571(state=PENDING, result=null, callback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773), errback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773)), so the following Deferred is getting paused: Deferred@463362299(state=PAUSED, result=Deferred@404161571, callback=, errback=) 00:27:30.059 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295 returned Deferred@404161571(state=PENDING, result=null, callback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295), errback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295)), so the following Deferred is getting paused: Deferred@1544825495(state=PAUSED, result=Deferred@404161571, callback=, errback=) 00:27:30.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.074766 23030 ts_tablet_manager.cc:1175] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5: Registered tablet (data state: TABLET_DATA_READY) 00:27:30.078 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485 returned Deferred@404161571(state=PENDING, result=null, callback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295) -> (continuation of Deferred@1308165263 after org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485), errback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295) -> (continuation of Deferred@1308165263 after org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485)), so the following Deferred is getting paused: Deferred@1308165263(state=PAUSED, result=Deferred@404161571, callback=, errback=) 00:27:30.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.080955 23265 ts_tablet_manager.cc:1030] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5: Bootstrapping tablet 00:27:30.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.082197 23265 tablet_bootstrap.cc:439] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5: Bootstrap starting. 00:27:30.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.079231 23031 ts_tablet_manager.cc:1175] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5: Registered tablet (data state: TABLET_DATA_READY) 00:27:30.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.088167 22896 ts_tablet_manager.cc:1175] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa: Registered tablet (data state: TABLET_DATA_READY) 00:27:30.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.090246 23265 tablet_bootstrap.cc:593] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5: No blocks or log segments found. Creating new log. 00:27:30.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.093979 23265 log.cc:525] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5: Log is configured to *not* fsync() on all Append() calls 00:27:30.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.088317 23028 ts_tablet_manager.cc:1175] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5: Registered tablet (data state: TABLET_DATA_READY) 00:27:30.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.097115 23029 ts_tablet_manager.cc:1175] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5: Registered tablet (data state: TABLET_DATA_READY) 00:27:30.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.108014 23265 tablet_bootstrap.cc:439] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5: No bootstrap required, opened a new log 00:27:30.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.109239 23265 ts_tablet_manager.cc:1047] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.004s 00:27:30.116 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@41be0726@1102972710 returned Deferred@404161571(state=PENDING, result=null, callback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295) -> (continuation of Deferred@1308165263 after org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485) -> (continuation of Deferred@1015270676 after org.apache.kudu.client.AsyncKuduClient$15@41be0726@1102972710), errback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295) -> (continuation of Deferred@1308165263 after org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485) -> (continuation of Deferred@1015270676 after org.apache.kudu.client.AsyncKuduClient$15@41be0726@1102972710)), so the following Deferred is getting paused: Deferred@1015270676(state=PAUSED, result=Deferred@404161571, callback=, errback=) 00:27:30.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.115175 23161 ts_tablet_manager.cc:1175] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f: Registered tablet (data state: TABLET_DATA_READY) 00:27:30.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.124320 23160 ts_tablet_manager.cc:1175] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f: Registered tablet (data state: TABLET_DATA_READY) 00:27:30.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.125455 23268 ts_tablet_manager.cc:1030] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f: Bootstrapping tablet 00:27:30.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.126154 23268 tablet_bootstrap.cc:439] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f: Bootstrap starting. 00:27:30.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.124910 23162 ts_tablet_manager.cc:1175] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f: Registered tablet (data state: TABLET_DATA_READY) 00:27:30.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.145362 23159 ts_tablet_manager.cc:1175] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f: Registered tablet (data state: TABLET_DATA_READY) 00:27:30.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.149791 23268 tablet_bootstrap.cc:593] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f: No blocks or log segments found. Creating new log. 00:27:30.155 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@566bf9d8@1449916888 returned Deferred@404161571(state=PENDING, result=null, callback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295) -> (continuation of Deferred@1308165263 after org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485) -> (continuation of Deferred@1015270676 after org.apache.kudu.client.AsyncKuduClient$15@41be0726@1102972710) -> (continuation of Deferred@1395872486 after org.apache.kudu.client.AsyncKuduClient$15@566bf9d8@1449916888), errback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295) -> (continuation of Deferred@1308165263 after org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485) -> (continuation of Deferred@1015270676 after org.apache.kudu.client.AsyncKuduClient$15@41be0726@1102972710) -> (continuation of Deferred@1395872486 after org.apache.kudu.client.AsyncKuduClient$15@566bf9d8@1449916888)), so the following Deferred is getting paused: Deferred@1395872486(state=PAUSED, result=Deferred@404161571, callback=, errback=) 00:27:30.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.161422 23268 log.cc:525] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f: Log is configured to *not* fsync() on all Append() calls 00:27:30.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.162187 23265 raft_consensus.cc:340] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.163167 23265 raft_consensus.cc:366] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:30.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.163986 23265 raft_consensus.cc:705] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1313c80b0ed640658101132ab87b8fc5, State: Initialized, Role: FOLLOWER 00:27:30.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.170430 23268 tablet_bootstrap.cc:439] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f: No bootstrap required, opened a new log 00:27:30.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.170861 23268 ts_tablet_manager.cc:1047] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.006s 00:27:30.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.176164 23265 consensus_queue.cc:229] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.026196 22895 tablet_service.cc:870] Processing CreateTablet for tablet fa4fe6266d3b4911aaac3bcebe15999b (table=testManyClients [id=f846a8dbe51f43e08571b061f393845f]), partition=RANGE (key) PARTITION 10 <= VALUES < 20 00:27:30.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.185158 22895 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fa4fe6266d3b4911aaac3bcebe15999b. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:30.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.036717 22894 tablet_service.cc:870] Processing CreateTablet for tablet d5cb2974aabf40dea3f204d4d5929375 (table=testManyClients [id=f846a8dbe51f43e08571b061f393845f]), partition=RANGE (key) PARTITION 20 <= VALUES < 30 00:27:30.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.202317 22894 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet d5cb2974aabf40dea3f204d4d5929375. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:30.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.203068 23265 ts_tablet_manager.cc:1075] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5: Time spent starting tablet: real 0.093s user 0.028s sys 0.000s 00:27:30.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.203490 23265 ts_tablet_manager.cc:1030] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5: Bootstrapping tablet 00:27:30.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.203867 23265 tablet_bootstrap.cc:439] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5: Bootstrap starting. 00:27:30.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.205322 23268 raft_consensus.cc:340] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.206264 23268 raft_consensus.cc:366] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:30.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.206485 23268 raft_consensus.cc:705] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ad861f849642ad801dde9f5e92c23f, State: Initialized, Role: FOLLOWER 00:27:30.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.207231 23268 consensus_queue.cc:229] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.209798 23225 heartbeater.cc:467] Master 127.22.19.252:32833 was elected leader, sending a full tablet report... 00:27:30.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.211649 23265 tablet_bootstrap.cc:593] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5: No blocks or log segments found. Creating new log. 00:27:30.219 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@6285f891@1652947089 returned Deferred@404161571(state=PENDING, result=null, callback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295) -> (continuation of Deferred@1308165263 after org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485) -> (continuation of Deferred@1015270676 after org.apache.kudu.client.AsyncKuduClient$15@41be0726@1102972710) -> (continuation of Deferred@1395872486 after org.apache.kudu.client.AsyncKuduClient$15@566bf9d8@1449916888) -> (continuation of Deferred@1087136440 after org.apache.kudu.client.AsyncKuduClient$15@6285f891@1652947089), errback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295) -> (continuation of Deferred@1308165263 after org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485) -> (continuation of Deferred@1015270676 after org.apache.kudu.client.AsyncKuduClient$15@41be0726@1102972710) -> (continuation of Deferred@1395872486 after org.apache.kudu.client.AsyncKuduClient$15@566bf9d8@1449916888) -> (continuation of Deferred@1087136440 after org.apache.kudu.client.AsyncKuduClient$15@6285f891@1652947089)), so the following Deferred is getting paused: Deferred@1087136440(state=PAUSED, result=Deferred@404161571, callback=, errback=) 00:27:30.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.039297 22893 tablet_service.cc:870] Processing CreateTablet for tablet 762f4bc1622449b8868b95d67f43e131 (table=testManyClients [id=f846a8dbe51f43e08571b061f393845f]), partition=RANGE (key) PARTITION VALUES >= 30 00:27:30.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.221230 22893 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 762f4bc1622449b8868b95d67f43e131. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:30.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.230702 22895 ts_tablet_manager.cc:1175] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa: Registered tablet (data state: TABLET_DATA_READY) 00:27:30.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.232692 22893 ts_tablet_manager.cc:1175] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa: Registered tablet (data state: TABLET_DATA_READY) 00:27:30.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.232509 23268 ts_tablet_manager.cc:1075] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f: Time spent starting tablet: real 0.061s user 0.018s sys 0.010s 00:27:30.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.234143 23268 ts_tablet_manager.cc:1030] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f: Bootstrapping tablet 00:27:30.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.234525 23268 tablet_bootstrap.cc:439] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f: Bootstrap starting. 00:27:30.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.235548 23266 ts_tablet_manager.cc:1030] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa: Bootstrapping tablet 00:27:30.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.236125 23266 tablet_bootstrap.cc:439] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa: Bootstrap starting. 00:27:30.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.237804 22894 ts_tablet_manager.cc:1175] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa: Registered tablet (data state: TABLET_DATA_READY) 00:27:30.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.238106 23265 tablet_bootstrap.cc:439] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5: No bootstrap required, opened a new log 00:27:30.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.238492 23265 ts_tablet_manager.cc:1047] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.000s 00:27:30.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.240203 23265 raft_consensus.cc:340] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.240878 23265 raft_consensus.cc:366] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:30.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.241086 23265 raft_consensus.cc:705] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1313c80b0ed640658101132ab87b8fc5, State: Initialized, Role: FOLLOWER 00:27:30.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.241720 23265 consensus_queue.cc:229] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.246428 23266 tablet_bootstrap.cc:593] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa: No blocks or log segments found. Creating new log. 00:27:30.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.246737 23268 tablet_bootstrap.cc:593] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f: No blocks or log segments found. Creating new log. 00:27:30.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.249603 23266 log.cc:525] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa: Log is configured to *not* fsync() on all Append() calls 00:27:30.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.251906 23270 raft_consensus.cc:472] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 00:27:30.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.252308 23270 raft_consensus.cc:494] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.254146 23270 leader_election.cc:296] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627), c9ad861f849642ad801dde9f5e92c23f (127.22.19.195:44599) 00:27:30.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.264257 23268 tablet_bootstrap.cc:439] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f: No bootstrap required, opened a new log 00:27:30.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.265578 23268 ts_tablet_manager.cc:1047] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.017s 00:27:30.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.267303 23266 tablet_bootstrap.cc:439] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa: No bootstrap required, opened a new log 00:27:30.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.267910 23266 ts_tablet_manager.cc:1047] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.012s 00:27:30.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.268968 23265 ts_tablet_manager.cc:1075] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s 00:27:30.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.269357 23265 ts_tablet_manager.cc:1030] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5: Bootstrapping tablet 00:27:30.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.269723 23265 tablet_bootstrap.cc:439] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5: Bootstrap starting. 00:27:30.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.276221 23268 raft_consensus.cc:340] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.276903 23268 raft_consensus.cc:366] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:30.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.277115 23268 raft_consensus.cc:705] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ad861f849642ad801dde9f5e92c23f, State: Initialized, Role: FOLLOWER 00:27:30.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.277747 23268 consensus_queue.cc:229] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.282841 23268 ts_tablet_manager.cc:1075] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s 00:27:30.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.283227 23268 ts_tablet_manager.cc:1030] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f: Bootstrapping tablet 00:27:30.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.283663 23268 tablet_bootstrap.cc:439] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f: Bootstrap starting. 00:27:30.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.288794 23268 tablet_bootstrap.cc:593] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f: No blocks or log segments found. Creating new log. 00:27:30.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.288874 23265 tablet_bootstrap.cc:593] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5: No blocks or log segments found. Creating new log. 00:27:30.297 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@4960f640@1231091264 returned Deferred@404161571(state=PENDING, result=null, callback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295) -> (continuation of Deferred@1308165263 after org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485) -> (continuation of Deferred@1015270676 after org.apache.kudu.client.AsyncKuduClient$15@41be0726@1102972710) -> (continuation of Deferred@1395872486 after org.apache.kudu.client.AsyncKuduClient$15@566bf9d8@1449916888) -> (continuation of Deferred@1087136440 after org.apache.kudu.client.AsyncKuduClient$15@6285f891@1652947089) -> (continuation of Deferred@9255400 after org.apache.kudu.client.AsyncKuduClient$15@4960f640@1231091264), errback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295) -> (continuation of Deferred@1308165263 after org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485) -> (continuation of Deferred@1015270676 after org.apache.kudu.client.AsyncKuduClient$15@41be0726@1102972710) -> (continuation of Deferred@1395872486 after org.apache.kudu.client.AsyncKuduClient$15@566bf9d8@1449916888) -> (continuation of Deferred@1087136440 after org.apache.kudu.client.AsyncKuduClient$15@6285f891@1652947089) -> (continuation of Deferred@9255400 after org.apache.kudu.client.AsyncKuduClient$15@4960f640@1231091264)), so the following Deferred is getting paused: Deferred@9255400(state=PAUSED, result=Deferred@404161571, callback=, errback=) 00:27:30.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.300173 23268 tablet_bootstrap.cc:439] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f: No bootstrap required, opened a new log 00:27:30.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.300530 23268 ts_tablet_manager.cc:1047] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.014s 00:27:30.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.302608 23268 raft_consensus.cc:340] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.303254 23268 raft_consensus.cc:366] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:30.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.304095 23268 raft_consensus.cc:705] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ad861f849642ad801dde9f5e92c23f, State: Initialized, Role: FOLLOWER 00:27:30.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.305960 23268 consensus_queue.cc:229] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.308019 22916 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "e3df67dd290b4bf6b4b9bc367348c886" candidate_uuid: "1313c80b0ed640658101132ab87b8fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab94ed628b14cf09ec9bd11be159caa" is_pre_election: true 00:27:30.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:30.310072 22985 leader_election.cc:348] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627): Illegal state: must be running to vote when last-logged opid is not known 00:27:30.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.317175 23265 tablet_bootstrap.cc:439] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5: No bootstrap required, opened a new log 00:27:30.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.317740 23265 ts_tablet_manager.cc:1047] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.004s 00:27:30.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.319598 23265 raft_consensus.cc:340] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.320399 23271 raft_consensus.cc:472] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 00:27:30.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.319712 23266 raft_consensus.cc:340] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.320750 23266 raft_consensus.cc:366] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:30.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.321020 23266 raft_consensus.cc:705] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab94ed628b14cf09ec9bd11be159caa, State: Initialized, Role: FOLLOWER 00:27:30.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.321499 23265 raft_consensus.cc:366] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:30.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.320807 23271 raft_consensus.cc:494] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.322273 23265 raft_consensus.cc:705] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1313c80b0ed640658101132ab87b8fc5, State: Initialized, Role: FOLLOWER 00:27:30.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.322223 23266 consensus_queue.cc:229] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.323658 23265 consensus_queue.cc:229] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:30.326578 23232 tablet.cc:1786] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f: Can't schedule compaction. Clean time has not been advanced past its initial value. 00:27:30.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.329423 23265 ts_tablet_manager.cc:1075] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s 00:27:30.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.330003 23271 leader_election.cc:296] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194:43751), 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627) 00:27:30.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.329931 23265 ts_tablet_manager.cc:1030] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5: Bootstrapping tablet 00:27:30.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.331768 23265 tablet_bootstrap.cc:439] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5: Bootstrap starting. 00:27:30.335 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@4d010818@1291913240 returned Deferred@404161571(state=PENDING, result=null, callback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295) -> (continuation of Deferred@1308165263 after org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485) -> (continuation of Deferred@1015270676 after org.apache.kudu.client.AsyncKuduClient$15@41be0726@1102972710) -> (continuation of Deferred@1395872486 after org.apache.kudu.client.AsyncKuduClient$15@566bf9d8@1449916888) -> (continuation of Deferred@1087136440 after org.apache.kudu.client.AsyncKuduClient$15@6285f891@1652947089) -> (continuation of Deferred@9255400 after org.apache.kudu.client.AsyncKuduClient$15@4960f640@1231091264) -> (continuation of Deferred@1861970594 after org.apache.kudu.client.AsyncKuduClient$15@4d010818@1291913240), errback=(continuation of Deferred@1033303349 after org.apache.kudu.client.AsyncKuduClient$2@4d468db4@1296469428) -> (continuation of Deferred@1229342093 after org.apache.kudu.client.AsyncKuduClient$15@528dd55a@1385026906) -> (continuation of Deferred@463362299 after org.apache.kudu.client.AsyncKuduClient$15@3b8875b5@998798773) -> (continuation of Deferred@1544825495 after org.apache.kudu.client.AsyncKuduClient$15@270579d7@654670295) -> (continuation of Deferred@1308165263 after org.apache.kudu.client.AsyncKuduClient$15@2449c97d@608815485) -> (continuation of Deferred@1015270676 after org.apache.kudu.client.AsyncKuduClient$15@41be0726@1102972710) -> (continuation of Deferred@1395872486 after org.apache.kudu.client.AsyncKuduClient$15@566bf9d8@1449916888) -> (continuation of Deferred@1087136440 after org.apache.kudu.client.AsyncKuduClient$15@6285f891@1652947089) -> (continuation of Deferred@9255400 after org.apache.kudu.client.AsyncKuduClient$15@4960f640@1231091264) -> (continuation of Deferred@1861970594 after org.apache.kudu.client.AsyncKuduClient$15@4d010818@1291913240)), so the following Deferred is getting paused: Deferred@1861970594(state=PAUSED, result=Deferred@404161571, callback=, errback=) 00:27:30.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.336125 23266 ts_tablet_manager.cc:1075] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa: Time spent starting tablet: real 0.068s user 0.021s sys 0.009s 00:27:30.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.336549 23266 ts_tablet_manager.cc:1030] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa: Bootstrapping tablet 00:27:30.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.337072 23266 tablet_bootstrap.cc:439] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa: Bootstrap starting. 00:27:30.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.340430 23265 tablet_bootstrap.cc:593] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5: No blocks or log segments found. Creating new log. 00:27:30.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.341745 23182 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "e3df67dd290b4bf6b4b9bc367348c886" candidate_uuid: "1313c80b0ed640658101132ab87b8fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ad861f849642ad801dde9f5e92c23f" is_pre_election: true 00:27:30.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.342886 23182 raft_consensus.cc:2346] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1313c80b0ed640658101132ab87b8fc5 in term 0. 00:27:30.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.344130 22985 leader_election.cc:310] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1313c80b0ed640658101132ab87b8fc5, c9ad861f849642ad801dde9f5e92c23f; no voters: 5ab94ed628b14cf09ec9bd11be159caa 00:27:30.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.346452 23270 raft_consensus.cc:2640] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Leader pre-election won for term 1 00:27:30.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.346704 23270 raft_consensus.cc:472] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 00:27:30.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.346946 23270 raft_consensus.cc:2886] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Advancing to term 1 00:27:30.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.350136 23265 tablet_bootstrap.cc:439] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5: No bootstrap required, opened a new log 00:27:30.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.350553 23265 ts_tablet_manager.cc:1047] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s 00:27:30.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.352231 23265 raft_consensus.cc:340] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.353067 23265 raft_consensus.cc:366] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:30.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.353271 23265 raft_consensus.cc:705] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1313c80b0ed640658101132ab87b8fc5, State: Initialized, Role: FOLLOWER 00:27:30.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.353772 23265 consensus_queue.cc:229] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.359923 23266 tablet_bootstrap.cc:593] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa: No blocks or log segments found. Creating new log. 00:27:30.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.364861 23265 ts_tablet_manager.cc:1075] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5: Time spent starting tablet: real 0.014s user 0.005s sys 0.004s 00:27:30.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.366042 23268 ts_tablet_manager.cc:1075] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f: Time spent starting tablet: real 0.065s user 0.003s sys 0.004s 00:27:30.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.366417 23268 ts_tablet_manager.cc:1030] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f: Bootstrapping tablet 00:27:30.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.366801 23268 tablet_bootstrap.cc:439] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f: Bootstrap starting. 00:27:30.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.352205 23270 raft_consensus.cc:494] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.369665 23270 leader_election.cc:296] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [CANDIDATE]: Term 1 election: Requested vote from peers 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627), c9ad861f849642ad801dde9f5e92c23f (127.22.19.195:44599) 00:27:30.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.370995 23266 tablet_bootstrap.cc:439] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa: No bootstrap required, opened a new log 00:27:30.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.371485 22916 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "e3df67dd290b4bf6b4b9bc367348c886" candidate_uuid: "1313c80b0ed640658101132ab87b8fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab94ed628b14cf09ec9bd11be159caa" 00:27:30.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.371835 23182 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "e3df67dd290b4bf6b4b9bc367348c886" candidate_uuid: "1313c80b0ed640658101132ab87b8fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ad861f849642ad801dde9f5e92c23f" 00:27:30.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.372040 22916 raft_consensus.cc:2886] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Advancing to term 1 00:27:30.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.372344 23182 raft_consensus.cc:2886] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Advancing to term 1 00:27:30.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.374492 23266 ts_tablet_manager.cc:1047] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.008s 00:27:30.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.378232 23266 raft_consensus.cc:340] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.379042 23266 raft_consensus.cc:366] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:30.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.379274 23266 raft_consensus.cc:705] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab94ed628b14cf09ec9bd11be159caa, State: Initialized, Role: FOLLOWER 00:27:30.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.380159 22916 raft_consensus.cc:2346] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1313c80b0ed640658101132ab87b8fc5 in term 1. 00:27:30.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.380465 22915 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "e3df67dd290b4bf6b4b9bc367348c886" candidate_uuid: "c9ad861f849642ad801dde9f5e92c23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab94ed628b14cf09ec9bd11be159caa" is_pre_election: true 00:27:30.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.381211 22915 raft_consensus.cc:2276] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9ad861f849642ad801dde9f5e92c23f in current term 1: Already voted for candidate 1313c80b0ed640658101132ab87b8fc5 in this term. 00:27:30.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.381422 22985 leader_election.cc:310] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1313c80b0ed640658101132ab87b8fc5, 5ab94ed628b14cf09ec9bd11be159caa; no voters: 00:27:30.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.382302 23283 raft_consensus.cc:2640] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [term 1 FOLLOWER]: Leader election won for term 1 00:27:30.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.383077 23283 raft_consensus.cc:667] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [term 1 LEADER]: Becoming Leader. State: Replica: 1313c80b0ed640658101132ab87b8fc5, State: Running, Role: LEADER 00:27:30.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.381768 23266 consensus_queue.cc:229] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.385361 23266 ts_tablet_manager.cc:1075] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s 00:27:30.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.385730 23266 ts_tablet_manager.cc:1030] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa: Bootstrapping tablet 00:27:30.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.386135 23266 tablet_bootstrap.cc:439] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa: Bootstrap starting. 00:27:30.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.384752 23283 consensus_queue.cc:206] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.389062 23266 tablet_bootstrap.cc:593] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa: No blocks or log segments found. Creating new log. 00:27:30.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.391417 23182 raft_consensus.cc:2346] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1313c80b0ed640658101132ab87b8fc5 in term 1. 00:27:30.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.394026 23266 tablet_bootstrap.cc:439] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa: No bootstrap required, opened a new log 00:27:30.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.394317 23266 ts_tablet_manager.cc:1047] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s 00:27:30.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.395464 23266 raft_consensus.cc:340] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.396051 23266 raft_consensus.cc:366] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:30.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.396252 23266 raft_consensus.cc:705] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab94ed628b14cf09ec9bd11be159caa, State: Initialized, Role: FOLLOWER 00:27:30.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.396783 23266 consensus_queue.cc:229] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.403180 23266 ts_tablet_manager.cc:1075] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa: Time spent starting tablet: real 0.009s user 0.004s sys 0.001s 00:27:30.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.403581 23266 ts_tablet_manager.cc:1030] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa: Bootstrapping tablet 00:27:30.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.403954 23266 tablet_bootstrap.cc:439] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa: Bootstrap starting. 00:27:30.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.407146 23266 tablet_bootstrap.cc:593] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa: No blocks or log segments found. Creating new log. 00:27:30.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.409394 23270 raft_consensus.cc:472] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 00:27:30.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.409771 23270 raft_consensus.cc:494] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.411267 23270 leader_election.cc:296] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627), c9ad861f849642ad801dde9f5e92c23f (127.22.19.195:44599) 00:27:30.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.413265 22915 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "762f4bc1622449b8868b95d67f43e131" candidate_uuid: "1313c80b0ed640658101132ab87b8fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab94ed628b14cf09ec9bd11be159caa" is_pre_election: true 00:27:30.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.413522 23266 tablet_bootstrap.cc:439] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa: No bootstrap required, opened a new log 00:27:30.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.413511 23182 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "762f4bc1622449b8868b95d67f43e131" candidate_uuid: "1313c80b0ed640658101132ab87b8fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ad861f849642ad801dde9f5e92c23f" is_pre_election: true 00:27:30.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.413870 23266 ts_tablet_manager.cc:1047] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s 00:27:30.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.413817 22915 raft_consensus.cc:2346] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1313c80b0ed640658101132ab87b8fc5 in term 0. 00:27:30.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.415213 23268 tablet_bootstrap.cc:593] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f: No blocks or log segments found. Creating new log. 00:27:30.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:30.415562 22985 leader_election.cc:348] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9ad861f849642ad801dde9f5e92c23f (127.22.19.195:44599): Illegal state: must be running to vote when last-logged opid is not known 00:27:30.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.415534 23266 raft_consensus.cc:340] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.416199 23266 raft_consensus.cc:366] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:30.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.416162 22985 leader_election.cc:310] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1313c80b0ed640658101132ab87b8fc5, 5ab94ed628b14cf09ec9bd11be159caa; no voters: c9ad861f849642ad801dde9f5e92c23f 00:27:30.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.416405 23266 raft_consensus.cc:705] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab94ed628b14cf09ec9bd11be159caa, State: Initialized, Role: FOLLOWER 00:27:30.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.417052 23270 raft_consensus.cc:2640] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Leader pre-election won for term 1 00:27:30.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.417313 23270 raft_consensus.cc:472] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 00:27:30.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.417548 23270 raft_consensus.cc:2886] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Advancing to term 1 00:27:30.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.416896 23266 consensus_queue.cc:229] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.420377 23266 ts_tablet_manager.cc:1075] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s 00:27:30.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.420708 23051 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "e3df67dd290b4bf6b4b9bc367348c886" candidate_uuid: "c9ad861f849642ad801dde9f5e92c23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1313c80b0ed640658101132ab87b8fc5" is_pre_election: true 00:27:30.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.421569 23051 raft_consensus.cc:2307] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c9ad861f849642ad801dde9f5e92c23f for term 1 because replica is either leader or believes a valid leader to be alive. 00:27:30.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.424060 23116 leader_election.cc:310] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9ad861f849642ad801dde9f5e92c23f; no voters: 1313c80b0ed640658101132ab87b8fc5, 5ab94ed628b14cf09ec9bd11be159caa 00:27:30.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:30.424300 23102 tablet.cc:1786] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5: Can't schedule compaction. Clean time has not been advanced past its initial value. 00:27:30.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.424820 23271 raft_consensus.cc:2588] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority 00:27:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.426244 22774 catalog_manager.cc:3791] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 reported cstate change: term changed from 0 to 1, leader changed from to 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194). New cstate: current_term: 1 leader_uuid: "1313c80b0ed640658101132ab87b8fc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } health_report { overall_health: UNKNOWN } } } 00:27:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.435485 23270 raft_consensus.cc:494] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.437865 22915 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "762f4bc1622449b8868b95d67f43e131" candidate_uuid: "1313c80b0ed640658101132ab87b8fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab94ed628b14cf09ec9bd11be159caa" 00:27:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.438369 23270 leader_election.cc:296] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [CANDIDATE]: Term 1 election: Requested vote from peers 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627), c9ad861f849642ad801dde9f5e92c23f (127.22.19.195:44599) 00:27:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.438374 22915 raft_consensus.cc:2886] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Advancing to term 1 00:27:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.438876 23182 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "762f4bc1622449b8868b95d67f43e131" candidate_uuid: "1313c80b0ed640658101132ab87b8fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ad861f849642ad801dde9f5e92c23f" 00:27:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.439463 23268 tablet_bootstrap.cc:439] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f: No bootstrap required, opened a new log 00:27:30.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.439805 23268 ts_tablet_manager.cc:1047] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.007s 00:27:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:30.440165 22985 leader_election.cc:348] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c9ad861f849642ad801dde9f5e92c23f (127.22.19.195:44599): Illegal state: must be running to vote when last-logged opid is not known 00:27:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.441485 23268 raft_consensus.cc:340] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.442226 23268 raft_consensus.cc:366] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.442438 23268 raft_consensus.cc:705] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ad861f849642ad801dde9f5e92c23f, State: Initialized, Role: FOLLOWER 00:27:30.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.442916 23268 consensus_queue.cc:229] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.449582 22915 raft_consensus.cc:2346] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1313c80b0ed640658101132ab87b8fc5 in term 1. 00:27:30.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.450634 22985 leader_election.cc:310] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1313c80b0ed640658101132ab87b8fc5, 5ab94ed628b14cf09ec9bd11be159caa; no voters: c9ad861f849642ad801dde9f5e92c23f 00:27:30.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.451298 23270 raft_consensus.cc:2640] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [term 1 FOLLOWER]: Leader election won for term 1 00:27:30.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.451709 23270 raft_consensus.cc:667] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [term 1 LEADER]: Becoming Leader. State: Replica: 1313c80b0ed640658101132ab87b8fc5, State: Running, Role: LEADER 00:27:30.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.452162 23270 consensus_queue.cc:206] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.463202 22774 catalog_manager.cc:3791] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 reported cstate change: term changed from 0 to 1, leader changed from to 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194). New cstate: current_term: 1 leader_uuid: "1313c80b0ed640658101132ab87b8fc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } health_report { overall_health: UNKNOWN } } } 00:27:30.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.464045 23268 ts_tablet_manager.cc:1075] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s 00:27:30.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:30.470232 22971 tablet.cc:1786] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa: Can't schedule compaction. Clean time has not been advanced past its initial value. 00:27:30.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.564416 23271 raft_consensus.cc:472] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 00:27:30.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.564731 23271 raft_consensus.cc:494] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.565789 23271 leader_election.cc:296] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627), 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194:43751) 00:27:30.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.566951 22915 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "d5cb2974aabf40dea3f204d4d5929375" candidate_uuid: "c9ad861f849642ad801dde9f5e92c23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab94ed628b14cf09ec9bd11be159caa" is_pre_election: true 00:27:30.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.567081 23051 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "d5cb2974aabf40dea3f204d4d5929375" candidate_uuid: "c9ad861f849642ad801dde9f5e92c23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1313c80b0ed640658101132ab87b8fc5" is_pre_election: true 00:27:30.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.567585 22915 raft_consensus.cc:2346] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ad861f849642ad801dde9f5e92c23f in term 0. 00:27:30.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.567685 23051 raft_consensus.cc:2346] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ad861f849642ad801dde9f5e92c23f in term 0. 00:27:30.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.568552 23116 leader_election.cc:310] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ab94ed628b14cf09ec9bd11be159caa, c9ad861f849642ad801dde9f5e92c23f; no voters: 00:27:30.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.569100 23271 raft_consensus.cc:2640] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Leader pre-election won for term 1 00:27:30.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.569309 23271 raft_consensus.cc:472] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 00:27:30.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.569535 23271 raft_consensus.cc:2886] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Advancing to term 1 00:27:30.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.572805 23271 raft_consensus.cc:494] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.573825 23271 leader_election.cc:296] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [CANDIDATE]: Term 1 election: Requested vote from peers 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627), 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194:43751) 00:27:30.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.574683 22915 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "d5cb2974aabf40dea3f204d4d5929375" candidate_uuid: "c9ad861f849642ad801dde9f5e92c23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab94ed628b14cf09ec9bd11be159caa" 00:27:30.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.574807 23051 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "d5cb2974aabf40dea3f204d4d5929375" candidate_uuid: "c9ad861f849642ad801dde9f5e92c23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1313c80b0ed640658101132ab87b8fc5" 00:27:30.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.575209 22915 raft_consensus.cc:2886] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Advancing to term 1 00:27:30.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.575383 23051 raft_consensus.cc:2886] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Advancing to term 1 00:27:30.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.578822 23279 raft_consensus.cc:472] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 00:27:30.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.579145 23279 raft_consensus.cc:494] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.580590 22915 raft_consensus.cc:2346] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ad861f849642ad801dde9f5e92c23f in term 1. 00:27:30.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.580826 23279 leader_election.cc:296] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194:43751), c9ad861f849642ad801dde9f5e92c23f (127.22.19.195:44599) 00:27:30.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.581480 23051 raft_consensus.cc:2346] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ad861f849642ad801dde9f5e92c23f in term 1. 00:27:30.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.582398 23116 leader_election.cc:310] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1313c80b0ed640658101132ab87b8fc5, c9ad861f849642ad801dde9f5e92c23f; no voters: 00:27:30.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.583305 23271 raft_consensus.cc:2640] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [term 1 FOLLOWER]: Leader election won for term 1 00:27:30.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.585808 23271 raft_consensus.cc:667] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [term 1 LEADER]: Becoming Leader. State: Replica: c9ad861f849642ad801dde9f5e92c23f, State: Running, Role: LEADER 00:27:30.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.586398 23271 consensus_queue.cc:206] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.594919 22774 catalog_manager.cc:3791] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f reported cstate change: term changed from 0 to 1, leader changed from to c9ad861f849642ad801dde9f5e92c23f (127.22.19.195). New cstate: current_term: 1 leader_uuid: "c9ad861f849642ad801dde9f5e92c23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } health_report { overall_health: HEALTHY } } } 00:27:30.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.598353 23271 raft_consensus.cc:472] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 00:27:30.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.598693 23271 raft_consensus.cc:494] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.600020 23051 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "fa4fe6266d3b4911aaac3bcebe15999b" candidate_uuid: "5ab94ed628b14cf09ec9bd11be159caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1313c80b0ed640658101132ab87b8fc5" is_pre_election: true 00:27:30.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.600564 23051 raft_consensus.cc:2346] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ab94ed628b14cf09ec9bd11be159caa in term 0. 00:27:30.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.601752 23271 leader_election.cc:296] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194:43751), 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627) 00:27:30.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.602332 23050 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "fa4fe6266d3b4911aaac3bcebe15999b" candidate_uuid: "c9ad861f849642ad801dde9f5e92c23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1313c80b0ed640658101132ab87b8fc5" is_pre_election: true 00:27:30.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.604348 22844 leader_election.cc:310] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1313c80b0ed640658101132ab87b8fc5, 5ab94ed628b14cf09ec9bd11be159caa; no voters: 00:27:30.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.605171 23279 raft_consensus.cc:2640] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Leader pre-election won for term 1 00:27:30.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.605429 23279 raft_consensus.cc:472] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 00:27:30.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.605679 23279 raft_consensus.cc:2886] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 0 FOLLOWER]: Advancing to term 1 00:27:30.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.602905 23050 raft_consensus.cc:2346] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ad861f849642ad801dde9f5e92c23f in term 0. 00:27:30.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.607417 23116 leader_election.cc:310] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1313c80b0ed640658101132ab87b8fc5, c9ad861f849642ad801dde9f5e92c23f; no voters: 00:27:30.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.608237 23271 raft_consensus.cc:2640] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Leader pre-election won for term 1 00:27:30.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.610404 23271 raft_consensus.cc:472] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 00:27:30.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.610725 23271 raft_consensus.cc:2886] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Advancing to term 1 00:27:30.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.612139 23279 raft_consensus.cc:494] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.613644 23279 leader_election.cc:296] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [CANDIDATE]: Term 1 election: Requested vote from peers 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194:43751), c9ad861f849642ad801dde9f5e92c23f (127.22.19.195:44599) 00:27:30.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.615326 23271 raft_consensus.cc:494] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.616606 22915 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "fa4fe6266d3b4911aaac3bcebe15999b" candidate_uuid: "c9ad861f849642ad801dde9f5e92c23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab94ed628b14cf09ec9bd11be159caa" is_pre_election: true 00:27:30.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.617251 22915 raft_consensus.cc:2276] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9ad861f849642ad801dde9f5e92c23f in current term 1: Already voted for candidate 5ab94ed628b14cf09ec9bd11be159caa in this term. 00:27:30.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.618409 23050 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "fa4fe6266d3b4911aaac3bcebe15999b" candidate_uuid: "c9ad861f849642ad801dde9f5e92c23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1313c80b0ed640658101132ab87b8fc5" 00:27:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.618883 23050 raft_consensus.cc:2886] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5 [term 0 FOLLOWER]: Advancing to term 1 00:27:30.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.618719 23051 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "fa4fe6266d3b4911aaac3bcebe15999b" candidate_uuid: "5ab94ed628b14cf09ec9bd11be159caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1313c80b0ed640658101132ab87b8fc5" 00:27:30.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.622833 23050 raft_consensus.cc:2346] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ad861f849642ad801dde9f5e92c23f in term 1. 00:27:30.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.623234 23051 raft_consensus.cc:2321] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ab94ed628b14cf09ec9bd11be159caa for term 1 because replica is already servicing an update from a current leader or another vote. 00:27:30.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.623914 23271 leader_election.cc:296] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [CANDIDATE]: Term 1 election: Requested vote from peers 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194:43751), 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627) 00:27:30.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.624538 23116 leader_election.cc:310] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1313c80b0ed640658101132ab87b8fc5, c9ad861f849642ad801dde9f5e92c23f; no voters: 00:27:30.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.625356 22915 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "fa4fe6266d3b4911aaac3bcebe15999b" candidate_uuid: "c9ad861f849642ad801dde9f5e92c23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab94ed628b14cf09ec9bd11be159caa" 00:27:30.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.625481 23271 raft_consensus.cc:2640] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 1 FOLLOWER]: Leader election won for term 1 00:27:30.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.625973 22915 raft_consensus.cc:2276] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c9ad861f849642ad801dde9f5e92c23f in current term 1: Already voted for candidate 5ab94ed628b14cf09ec9bd11be159caa in this term. 00:27:30.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.626507 23271 raft_consensus.cc:667] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 1 LEADER]: Becoming Leader. State: Replica: c9ad861f849642ad801dde9f5e92c23f, State: Running, Role: LEADER 00:27:30.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.627064 23271 consensus_queue.cc:206] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.627310 23182 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "fa4fe6266d3b4911aaac3bcebe15999b" candidate_uuid: "5ab94ed628b14cf09ec9bd11be159caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ad861f849642ad801dde9f5e92c23f" is_pre_election: true 00:27:30.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.628068 23181 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "fa4fe6266d3b4911aaac3bcebe15999b" candidate_uuid: "5ab94ed628b14cf09ec9bd11be159caa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ad861f849642ad801dde9f5e92c23f" 00:27:30.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.631063 23182 raft_consensus.cc:2307] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 5ab94ed628b14cf09ec9bd11be159caa for term 1 because replica is either leader or believes a valid leader to be alive. 00:27:30.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.631754 23181 raft_consensus.cc:2321] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [term 1 LEADER]: Leader election vote request: Denying vote to candidate 5ab94ed628b14cf09ec9bd11be159caa for term 1 because replica is already servicing an update from a current leader or another vote. 00:27:30.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.633275 22774 catalog_manager.cc:3791] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f reported cstate change: term changed from 0 to 1, leader changed from to c9ad861f849642ad801dde9f5e92c23f (127.22.19.195). New cstate: current_term: 1 leader_uuid: "c9ad861f849642ad801dde9f5e92c23f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } health_report { overall_health: HEALTHY } } } 00:27:30.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.636029 22844 leader_election.cc:310] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ab94ed628b14cf09ec9bd11be159caa; no voters: 1313c80b0ed640658101132ab87b8fc5, c9ad861f849642ad801dde9f5e92c23f 00:27:30.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.636871 23279 raft_consensus.cc:2588] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority 00:27:30.677 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:30.677 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:30.678 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:30.680 [DEBUG - New I/O worker #1] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299998, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299998, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299998, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:30.682 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, DeadlineTracker(timeout=30000, elapsed=20), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.22.19.252:32833, [6ms] Sub RPC GetTableLocations: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:30.687 [DEBUG - New I/O worker #1] (Deferred.java:1330) callback=retry RPC@317421085 returned Deferred@1930707263(state=PENDING, result=null, callback=(continuation of Deferred@535259753 after retry RPC@317421085), errback=(continuation of Deferred@535259753 after retry RPC@317421085)), so the following Deferred is getting paused: Deferred@535259753(state=PAUSED, result=Deferred@1930707263, callback=org.apache.kudu.client.AsyncKuduSession$2@1520be64 -> passthrough -> wakeup thread Time-limited test, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@5c326943 -> wakeup thread Time-limited test) 00:27:30.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.708400 23271 raft_consensus.cc:472] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 00:27:30.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.708827 23271 raft_consensus.cc:494] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 } } peers { permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 } } peers { permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 } } 00:27:30.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.710077 23271 leader_election.cc:296] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1313c80b0ed640658101132ab87b8fc5 (127.22.19.194:43751), 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627) 00:27:30.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.711453 23051 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "762f4bc1622449b8868b95d67f43e131" candidate_uuid: "c9ad861f849642ad801dde9f5e92c23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1313c80b0ed640658101132ab87b8fc5" is_pre_election: true 00:27:30.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.712301 23051 raft_consensus.cc:2307] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate c9ad861f849642ad801dde9f5e92c23f for term 1 because replica is either leader or believes a valid leader to be alive. 00:27:30.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.712512 22915 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "762f4bc1622449b8868b95d67f43e131" candidate_uuid: "c9ad861f849642ad801dde9f5e92c23f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ab94ed628b14cf09ec9bd11be159caa" is_pre_election: true 00:27:30.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.713413 22915 raft_consensus.cc:2276] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c9ad861f849642ad801dde9f5e92c23f in current term 1: Already voted for candidate 1313c80b0ed640658101132ab87b8fc5 in this term. 00:27:30.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.714931 23116 leader_election.cc:310] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9ad861f849642ad801dde9f5e92c23f; no voters: 1313c80b0ed640658101132ab87b8fc5, 5ab94ed628b14cf09ec9bd11be159caa 00:27:30.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.715824 23271 raft_consensus.cc:2886] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f [term 0 FOLLOWER]: Advancing to term 1 00:27:30.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.718685 23271 raft_consensus.cc:2588] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority 00:27:30.791 [DEBUG - New I/O worker #4] (Negotiator.java:764) Authenticated connection [id: 0x6e94da57, /127.0.0.1:35292 => /127.22.19.195:44599] using TOKEN/null 00:27:30.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.802614 22915 raft_consensus.cc:1181] T fa4fe6266d3b4911aaac3bcebe15999b P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Refusing update from remote peer c9ad861f849642ad801dde9f5e92c23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 00:27:30.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.802700 23051 raft_consensus.cc:1181] T fa4fe6266d3b4911aaac3bcebe15999b P 1313c80b0ed640658101132ab87b8fc5 [term 1 FOLLOWER]: Refusing update from remote peer c9ad861f849642ad801dde9f5e92c23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 00:27:30.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.804061 23292 consensus_queue.cc:984] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 00:27:30.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.804883 23271 consensus_queue.cc:984] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 00:27:30.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.825157 23296 mvcc.cc:203] Tried to move safe_time back from 6351349558476218368 to 6351349557774643200. Current Snapshot: MvccSnapshot[committed={T|T < 6351349558476218368}] 00:27:30.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.833485 23297 mvcc.cc:203] Tried to move safe_time back from 6351349558476218368 to 6351349557774643200. Current Snapshot: MvccSnapshot[committed={T|T < 6351349558476218368}] 00:27:30.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.855435 23181 raft_consensus.cc:1181] T e3df67dd290b4bf6b4b9bc367348c886 P c9ad861f849642ad801dde9f5e92c23f [term 1 FOLLOWER]: Refusing update from remote peer 1313c80b0ed640658101132ab87b8fc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 00:27:30.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.858901 23270 consensus_queue.cc:984] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 00:27:30.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.880663 22916 raft_consensus.cc:1181] T e3df67dd290b4bf6b4b9bc367348c886 P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Refusing update from remote peer 1313c80b0ed640658101132ab87b8fc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 00:27:30.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.883087 23270 consensus_queue.cc:984] T e3df67dd290b4bf6b4b9bc367348c886 P 1313c80b0ed640658101132ab87b8fc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 00:27:30.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.919452 22915 raft_consensus.cc:1181] T d5cb2974aabf40dea3f204d4d5929375 P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Refusing update from remote peer c9ad861f849642ad801dde9f5e92c23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 00:27:30.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.919772 23050 raft_consensus.cc:1181] T d5cb2974aabf40dea3f204d4d5929375 P 1313c80b0ed640658101132ab87b8fc5 [term 1 FOLLOWER]: Refusing update from remote peer c9ad861f849642ad801dde9f5e92c23f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 00:27:30.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.921717 23305 consensus_queue.cc:984] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s 00:27:30.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.922255 23303 consensus_queue.cc:984] T d5cb2974aabf40dea3f204d4d5929375 P c9ad861f849642ad801dde9f5e92c23f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1313c80b0ed640658101132ab87b8fc5" member_type: VOTER last_known_addr { host: "127.22.19.194" port: 43751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 00:27:30.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.972317 23181 raft_consensus.cc:1181] T 762f4bc1622449b8868b95d67f43e131 P c9ad861f849642ad801dde9f5e92c23f [term 1 FOLLOWER]: Refusing update from remote peer 1313c80b0ed640658101132ab87b8fc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 00:27:30.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.974788 23283 consensus_queue.cc:984] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ad861f849642ad801dde9f5e92c23f" member_type: VOTER last_known_addr { host: "127.22.19.195" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 00:27:30.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.998373 22915 raft_consensus.cc:1181] T 762f4bc1622449b8868b95d67f43e131 P 5ab94ed628b14cf09ec9bd11be159caa [term 1 FOLLOWER]: Refusing update from remote peer 1313c80b0ed640658101132ab87b8fc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 00:27:31.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:30.999778 23283 consensus_queue.cc:984] T 762f4bc1622449b8868b95d67f43e131 P 1313c80b0ed640658101132ab87b8fc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ab94ed628b14cf09ec9bd11be159caa" member_type: VOTER last_known_addr { host: "127.22.19.193" port: 35627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 00:27:31.043 [DEBUG - New I/O worker #5] (Negotiator.java:764) Authenticated connection [id: 0x30089f1a, /127.0.0.1:38914 => /127.22.19.194:43751] using TOKEN/null 00:27:31.378 [DEBUG - pool-3-thread-5] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.379 [DEBUG - pool-3-thread-17] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.483 [DEBUG - pool-3-thread-7] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.483 [DEBUG - pool-3-thread-67] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.482 [DEBUG - pool-3-thread-30] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.480 [DEBUG - pool-3-thread-9] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.478 [DEBUG - pool-3-thread-20] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.477 [DEBUG - pool-3-thread-15] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.472 [DEBUG - pool-3-thread-4] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.464 [DEBUG - pool-3-thread-32] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.464 [DEBUG - pool-3-thread-25] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.455 [DEBUG - pool-3-thread-18] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.452 [DEBUG - pool-3-thread-3] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.447 [DEBUG - pool-3-thread-16] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.426 [DEBUG - pool-3-thread-24] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.410 [DEBUG - pool-3-thread-26] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.406 [DEBUG - pool-3-thread-33] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.403 [DEBUG - pool-3-thread-6] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.395 [DEBUG - pool-3-thread-13] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.386 [DEBUG - pool-3-thread-21] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.380 [DEBUG - pool-3-thread-2] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.380 [DEBUG - pool-3-thread-8] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.379 [DEBUG - pool-3-thread-1] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.497 [DEBUG - pool-3-thread-25] (NetUtil.java:106) Resolved IP of `127.22.19.252' to /127.22.19.252 in 6063196ns 00:27:31.655 [DEBUG - New I/O worker #14] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.709 [DEBUG - New I/O worker #248] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.710 [DEBUG - New I/O worker #20] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.722 [DEBUG - pool-3-thread-22] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.727 [DEBUG - New I/O worker #12] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.728 [DEBUG - New I/O worker #89] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.729 [DEBUG - New I/O worker #37] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.730 [DEBUG - New I/O worker #47] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.732 [DEBUG - New I/O worker #53] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.734 [DEBUG - New I/O worker #127] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.734 [DEBUG - New I/O worker #43] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.735 [DEBUG - New I/O worker #86] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.735 [DEBUG - New I/O worker #58] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.737 [DEBUG - New I/O worker #16] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.738 [DEBUG - New I/O worker #23] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.739 [DEBUG - New I/O worker #85] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.741 [DEBUG - New I/O worker #39] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.743 [DEBUG - New I/O worker #106] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.757 [DEBUG - New I/O worker #41] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.756 [DEBUG - New I/O worker #80] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.756 [DEBUG - New I/O worker #117] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.755 [DEBUG - New I/O worker #24] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.754 [DEBUG - New I/O worker #251] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.754 [DEBUG - New I/O worker #133] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.753 [DEBUG - New I/O worker #84] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.752 [DEBUG - New I/O worker #122] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.752 [DEBUG - New I/O worker #91] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.743 [DEBUG - New I/O worker #107] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.751 [DEBUG - New I/O worker #301] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.750 [DEBUG - New I/O worker #75] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.749 [DEBUG - New I/O worker #21] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.746 [DEBUG - New I/O worker #119] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.745 [DEBUG - New I/O worker #29] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.744 [DEBUG - New I/O worker #124] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.744 [DEBUG - New I/O worker #36] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.778 [DEBUG - New I/O worker #113] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.788 [DEBUG - New I/O worker #123] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.790 [DEBUG - New I/O worker #38] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.791 [DEBUG - New I/O worker #60] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.792 [DEBUG - New I/O worker #114] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.795 [DEBUG - New I/O worker #69] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.800 [DEBUG - New I/O worker #45] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.806 [DEBUG - New I/O worker #105] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.808 [DEBUG - New I/O worker #82] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.808 [DEBUG - New I/O worker #18] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.809 [DEBUG - New I/O worker #25] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.810 [DEBUG - New I/O worker #10] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.814 [DEBUG - New I/O worker #71] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.816 [DEBUG - New I/O worker #19] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.818 [DEBUG - New I/O worker #78] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.821 [DEBUG - New I/O worker #15] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.823 [DEBUG - New I/O worker #30] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.825 [DEBUG - New I/O worker #49] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.827 [DEBUG - New I/O worker #112] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.830 [DEBUG - New I/O worker #48] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.832 [DEBUG - New I/O worker #79] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.840 [DEBUG - New I/O worker #11] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.840 [DEBUG - New I/O worker #13] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.911 [DEBUG - New I/O worker #108] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.912 [DEBUG - New I/O worker #46] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.912 [DEBUG - New I/O worker #185] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.913 [DEBUG - New I/O worker #74] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:31.924 [DEBUG - New I/O worker #111] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:31.926 [DEBUG - New I/O worker #67] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.942 [DEBUG - New I/O worker #22] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:31.943 [DEBUG - pool-3-thread-11] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.957 [DEBUG - pool-3-thread-29] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.960 [DEBUG - pool-3-thread-10] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.962 [DEBUG - pool-3-thread-34] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.963 [DEBUG - pool-3-thread-14] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.964 [DEBUG - pool-3-thread-37] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.965 [DEBUG - pool-3-thread-12] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.965 [DEBUG - pool-3-thread-28] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:31.996 [DEBUG - pool-3-thread-27] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.076 [DEBUG - pool-3-thread-70] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.077 [DEBUG - pool-3-thread-48] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.084 [DEBUG - pool-3-thread-44] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.087 [DEBUG - pool-3-thread-40] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.126 [DEBUG - New I/O worker #180] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.127 [DEBUG - New I/O worker #76] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.140 [DEBUG - New I/O worker #241] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.143 [DEBUG - New I/O worker #76] (Negotiator.java:764) Authenticated connection [id: 0xfe6aef1c, /127.0.0.1:37364 => /127.22.19.252:32833] using SASL/PLAIN 00:27:32.153 [DEBUG - New I/O worker #285] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.161 [DEBUG - New I/O worker #129] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.162 [DEBUG - New I/O worker #56] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.165 [DEBUG - New I/O worker #77] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.170 [DEBUG - New I/O worker #121] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.179 [DEBUG - New I/O worker #54] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.181 [DEBUG - New I/O worker #109] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.233 [DEBUG - New I/O worker #89] (Negotiator.java:764) Authenticated connection [id: 0x96d35ca4, /127.0.0.1:52602 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.242 [DEBUG - New I/O worker #178] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.295 [DEBUG - New I/O worker #280] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.301 [DEBUG - New I/O worker #183] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.302 [DEBUG - New I/O worker #146] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.302 [DEBUG - New I/O worker #193] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.305 [DEBUG - New I/O worker #217] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.314 [DEBUG - New I/O worker #64] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.324 [DEBUG - New I/O worker #126] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.327 [DEBUG - New I/O worker #27] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.328 [DEBUG - New I/O worker #103] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.329 [DEBUG - New I/O worker #181] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.330 [DEBUG - New I/O worker #50] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.333 [DEBUG - New I/O worker #83] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.339 [DEBUG - New I/O worker #28] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.353 [DEBUG - pool-3-thread-65] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.354 [DEBUG - pool-3-thread-62] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.355 [DEBUG - pool-3-thread-54] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.373 [DEBUG - pool-3-thread-19] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.374 [DEBUG - New I/O worker #35] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.374 [DEBUG - pool-3-thread-69] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.375 [DEBUG - New I/O worker #340] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.375 [DEBUG - New I/O worker #292] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.379 [DEBUG - New I/O worker #383] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.399 [DEBUG - New I/O worker #13] (Negotiator.java:764) Authenticated connection [id: 0xbb09437d, /127.0.0.1:37328 => /127.22.19.252:32833] using SASL/PLAIN 00:27:32.413 [DEBUG - pool-3-thread-43] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.414 [DEBUG - pool-3-thread-55] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.415 [DEBUG - New I/O worker #116] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.417 [DEBUG - New I/O worker #17] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.421 [DEBUG - New I/O worker #102] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.422 [DEBUG - New I/O worker #137] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.428 [DEBUG - pool-3-thread-64] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.437 [DEBUG - New I/O worker #91] (Negotiator.java:764) Authenticated connection [id: 0xe2bcfba8, /127.0.0.1:37250 => /127.22.19.252:32833] using SASL/PLAIN 00:27:32.444 [DEBUG - New I/O worker #42] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.445 [DEBUG - pool-3-thread-39] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.446 [DEBUG - pool-3-thread-23] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.455 [DEBUG - New I/O worker #113] (Negotiator.java:764) Authenticated connection [id: 0xc78bce22, /127.0.0.1:52656 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.468 [DEBUG - New I/O worker #80] (Negotiator.java:764) Authenticated connection [id: 0x7849b2c8, /127.0.0.1:52620 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.473 [DEBUG - pool-3-thread-35] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.481 [DEBUG - pool-3-thread-68] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.499 [DEBUG - New I/O worker #107] (Negotiator.java:764) Authenticated connection [id: 0xca247ca5, /127.0.0.1:39220 => /127.22.19.253:37879] using SASL/PLAIN 00:27:32.504 [DEBUG - New I/O worker #20] (Negotiator.java:764) Authenticated connection [id: 0xd6ae9b68, /127.0.0.1:52572 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.506 [DEBUG - New I/O worker #82] (Negotiator.java:764) Authenticated connection [id: 0x95623057, /127.0.0.1:52680 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.510 [DEBUG - New I/O worker #122] (Negotiator.java:764) Authenticated connection [id: 0xc5ea3a17, /127.0.0.1:39238 => /127.22.19.253:37879] using SASL/PLAIN 00:27:32.513 [DEBUG - New I/O worker #251] (Negotiator.java:764) Authenticated connection [id: 0xbc2dc9aa, /127.0.0.1:52578 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.514 [DEBUG - New I/O worker #21] (Negotiator.java:764) Authenticated connection [id: 0x17347ad5, /127.0.0.1:52586 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.523 [DEBUG - New I/O worker #36] (Negotiator.java:764) Authenticated connection [id: 0xa295c955, /127.0.0.1:52626 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.524 [DEBUG - New I/O worker #85] (Negotiator.java:764) Authenticated connection [id: 0x7c878b53, /127.0.0.1:52634 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.532 [DEBUG - New I/O worker #134] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.533 [DEBUG - New I/O worker #11] (Negotiator.java:764) Authenticated connection [id: 0xb5c6ca9c, /127.0.0.1:37268 => /127.22.19.252:32833] using SASL/PLAIN 00:27:32.535 [DEBUG - New I/O worker #19] (Negotiator.java:764) Authenticated connection [id: 0x971b4d64, /127.0.0.1:37304 => /127.22.19.252:32833] using SASL/PLAIN 00:27:32.548 [DEBUG - New I/O worker #46] (Negotiator.java:764) Authenticated connection [id: 0xd532b08e, /127.0.0.1:37346 => /127.22.19.252:32833] using SASL/PLAIN 00:27:32.583 [DEBUG - New I/O worker #340] (Negotiator.java:764) Authenticated connection [id: 0x31416408, /127.0.0.1:52780 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.593 [DEBUG - New I/O worker #38] (Negotiator.java:764) Authenticated connection [id: 0xbd6bdc53, /127.0.0.1:52636 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.595 [DEBUG - New I/O worker #92] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.624 [DEBUG - New I/O worker #28] (Negotiator.java:764) Authenticated connection [id: 0x1aeb561d, /127.0.0.1:37380 => /127.22.19.252:32833] using SASL/PLAIN 00:27:32.639 [DEBUG - New I/O worker #146] (Negotiator.java:764) Authenticated connection [id: 0x8be729ae, /127.0.0.1:52716 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.642 [DEBUG - New I/O worker #280] (Negotiator.java:764) Authenticated connection [id: 0xd4228449, /127.0.0.1:52770 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.501 [DEBUG - New I/O worker #29] (Negotiator.java:764) Authenticated connection [id: 0x186afacc, /127.0.0.1:39274 => /127.22.19.253:37879] using SASL/PLAIN 00:27:32.670 [DEBUG - New I/O worker #177] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.670 [DEBUG - New I/O worker #115] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.712 [DEBUG - New I/O worker #83] (Negotiator.java:764) Authenticated connection [id: 0x4bbc7519, /127.0.0.1:52746 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.718 [DEBUG - New I/O worker #177] (Negotiator.java:764) Authenticated connection [id: 0x525de309, /127.0.0.1:39412 => /127.22.19.253:37879] using SASL/PLAIN 00:27:32.721 [DEBUG - New I/O worker #45] (Negotiator.java:764) Authenticated connection [id: 0x47a083f9, /127.0.0.1:52664 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.723 [DEBUG - New I/O worker #51] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.724 [DEBUG - New I/O worker #389] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.728 [DEBUG - New I/O worker #130] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.729 [DEBUG - New I/O worker #306] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.730 [DEBUG - New I/O worker #120] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.733 [DEBUG - New I/O worker #72] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.733 [DEBUG - New I/O worker #175] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.734 [DEBUG - New I/O worker #405] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.736 [DEBUG - New I/O worker #93] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.737 [DEBUG - pool-3-thread-60] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.740 [DEBUG - pool-3-thread-61] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.744 [DEBUG - pool-3-thread-31] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.745 [DEBUG - pool-3-thread-36] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.746 [DEBUG - pool-3-thread-79] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.747 [DEBUG - pool-3-thread-59] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.749 [DEBUG - pool-3-thread-66] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.750 [DEBUG - pool-3-thread-38] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.752 [DEBUG - pool-3-thread-52] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.754 [DEBUG - pool-3-thread-53] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.754 [DEBUG - pool-3-thread-41] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.755 [DEBUG - pool-3-thread-63] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.756 [DEBUG - pool-3-thread-56] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.758 [DEBUG - pool-3-thread-42] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.759 [DEBUG - pool-3-thread-77] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.767 [DEBUG - pool-3-thread-72] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.769 [DEBUG - pool-3-thread-73] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.769 [DEBUG - pool-3-thread-47] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.769 [DEBUG - pool-3-thread-57] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.768 [DEBUG - pool-3-thread-49] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.768 [DEBUG - pool-3-thread-45] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.768 [DEBUG - pool-3-thread-58] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.767 [DEBUG - pool-3-thread-76] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.767 [DEBUG - New I/O worker #387] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.777 [DEBUG - New I/O worker #54] (Negotiator.java:764) Authenticated connection [id: 0xc2a22ee5, /127.0.0.1:52704 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.766 [DEBUG - pool-3-thread-50] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.766 [DEBUG - pool-3-thread-80] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.765 [DEBUG - pool-3-thread-71] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.765 [DEBUG - pool-3-thread-46] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.765 [DEBUG - pool-3-thread-74] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.782 [DEBUG - pool-3-thread-75] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:32.806 [DEBUG - New I/O worker #362] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.806 [DEBUG - New I/O worker #474] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.808 [DEBUG - New I/O worker #476] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.814 [DEBUG - New I/O worker #134] (Negotiator.java:764) Authenticated connection [id: 0xdb565a01, /127.0.0.1:37406 => /127.22.19.252:32833] using SASL/PLAIN 00:27:32.816 [DEBUG - New I/O worker #130] (Negotiator.java:764) Authenticated connection [id: 0x8a9100a9, /127.0.0.1:39394 => /127.22.19.253:37879] using SASL/PLAIN 00:27:32.823 [DEBUG - New I/O worker #258] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.825 [DEBUG - New I/O worker #265] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.826 [DEBUG - New I/O worker #413] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.830 [DEBUG - New I/O worker #259] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.832 [DEBUG - New I/O worker #415] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.842 [DEBUG - New I/O worker #298] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.842 [DEBUG - New I/O worker #221] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.847 [DEBUG - New I/O worker #268] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.847 [DEBUG - New I/O worker #219] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.848 [DEBUG - New I/O worker #357] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.865 [DEBUG - New I/O worker #220] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.871 [DEBUG - New I/O worker #337] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.872 [DEBUG - New I/O worker #357] (Negotiator.java:764) Authenticated connection [id: 0x86feb280, /127.0.0.1:39442 => /127.22.19.253:37879] using SASL/PLAIN 00:27:32.874 [DEBUG - New I/O worker #442] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.877 [DEBUG - New I/O worker #393] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.881 [DEBUG - New I/O worker #428] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.882 [DEBUG - New I/O worker #354] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.883 [DEBUG - New I/O worker #322] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:32.887 [DEBUG - New I/O worker #592] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.892 [DEBUG - New I/O worker #413] (Negotiator.java:764) Authenticated connection [id: 0x60d2acb7, /127.0.0.1:52806 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.904 [DEBUG - New I/O worker #289] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.907 [DEBUG - New I/O worker #592] (Negotiator.java:764) Authenticated connection [id: 0xd86e036a, /127.0.0.1:52842 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.910 [DEBUG - New I/O worker #348] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.912 [DEBUG - New I/O worker #365] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.921 [DEBUG - New I/O worker #393] (Negotiator.java:764) Authenticated connection [id: 0x08b597c4, /127.0.0.1:52824 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.926 [DEBUG - New I/O worker #300] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.948 [DEBUG - New I/O worker #528] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.948 [DEBUG - New I/O worker #345] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:32.952 [DEBUG - New I/O worker #637] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.956 [DEBUG - New I/O worker #428] (Negotiator.java:764) Authenticated connection [id: 0xc4d95fd5, /127.0.0.1:52836 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.960 [DEBUG - New I/O worker #289] (Negotiator.java:764) Authenticated connection [id: 0xe819caec, /127.0.0.1:52848 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.972 [DEBUG - New I/O worker #389] (Negotiator.java:764) Authenticated connection [id: 0xc8cf6784, /127.0.0.1:39388 => /127.22.19.253:37879] using SASL/PLAIN 00:27:32.977 [DEBUG - New I/O worker #298] (Negotiator.java:764) Authenticated connection [id: 0xc30948eb, /127.0.0.1:52818 => /127.22.19.254:44849] using SASL/PLAIN 00:27:32.989 [DEBUG - New I/O worker #390] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.989 [DEBUG - New I/O worker #374] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.983 [DEBUG - New I/O worker #376] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:32.978 [DEBUG - New I/O worker #418] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.005 [DEBUG - New I/O worker #273] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.026 [DEBUG - New I/O worker #258] (Negotiator.java:764) Authenticated connection [id: 0xf4577b8d, /127.0.0.1:52800 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.031 [DEBUG - New I/O worker #348] (Negotiator.java:764) Authenticated connection [id: 0x7e23c18f, /127.0.0.1:52854 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.004 [DEBUG - New I/O worker #300] (Negotiator.java:764) Authenticated connection [id: 0x6d26305a, /127.0.0.1:52860 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.003 [DEBUG - New I/O worker #475] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.001 [DEBUG - New I/O worker #362] (Negotiator.java:764) Authenticated connection [id: 0x789ce54a, /127.0.0.1:37448 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.038 [DEBUG - New I/O worker #275] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.039 [DEBUG - New I/O worker #324] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.039 [DEBUG - New I/O worker #260] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.053 [DEBUG - New I/O worker #354] (Negotiator.java:764) Authenticated connection [id: 0x446cdca3, /127.0.0.1:52830 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.054 [DEBUG - New I/O worker #51] (Negotiator.java:764) Authenticated connection [id: 0xe23e6695, /127.0.0.1:52786 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.054 [DEBUG - New I/O worker #350] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.056 [DEBUG - New I/O worker #269] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.058 [DEBUG - New I/O worker #141] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.066 [DEBUG - New I/O worker #270] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.285 [DEBUG - New I/O worker #415] (Negotiator.java:764) Authenticated connection [id: 0x71e17cf5, /127.0.0.1:39438 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.285 [DEBUG - New I/O worker #42] (Negotiator.java:764) Authenticated connection [id: 0xcc4ecd49, /127.0.0.1:52726 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.286 [DEBUG - New I/O worker #77] (Negotiator.java:764) Authenticated connection [id: 0x73fa576f, /127.0.0.1:39322 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.288 [DEBUG - New I/O worker #285] (Negotiator.java:764) Authenticated connection [id: 0x33b4cc54, /127.0.0.1:39346 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.290 [DEBUG - New I/O worker #337] (Negotiator.java:764) Authenticated connection [id: 0xd27e3497, /127.0.0.1:39448 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.299 [DEBUG - New I/O worker #352] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.300 [DEBUG - New I/O worker #345] (Negotiator.java:764) Authenticated connection [id: 0x3fe18519, /127.0.0.1:52866 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.301 [DEBUG - New I/O worker #637] (Negotiator.java:764) Authenticated connection [id: 0x86bcef08, /127.0.0.1:39472 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.303 [DEBUG - New I/O worker #178] (Negotiator.java:764) Authenticated connection [id: 0x06bb5a24, /127.0.0.1:39382 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.304 [DEBUG - New I/O worker #416] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.283 [DEBUG - New I/O worker #268] (Negotiator.java:764) Authenticated connection [id: 0xac01ca11, /127.0.0.1:37460 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.280 [DEBUG - New I/O worker #293] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.278 [DEBUG - New I/O worker #259] (Negotiator.java:764) Authenticated connection [id: 0xa8762379, /127.0.0.1:39430 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.278 [DEBUG - New I/O worker #290] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.277 [DEBUG - New I/O worker #402] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.276 [DEBUG - New I/O worker #342] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.313 [DEBUG - New I/O worker #272] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.276 [DEBUG - New I/O worker #81] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.315 [DEBUG - New I/O worker #425] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.276 [DEBUG - New I/O worker #322] (Negotiator.java:764) Authenticated connection [id: 0xdfeaae7b, /127.0.0.1:37478 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.275 [DEBUG - New I/O worker #267] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.274 [DEBUG - New I/O worker #219] (Negotiator.java:764) Authenticated connection [id: 0xa82700ce, /127.0.0.1:37472 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.271 [DEBUG - New I/O worker #407] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.270 [DEBUG - New I/O worker #142] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.270 [DEBUG - New I/O worker #405] (Negotiator.java:764) Authenticated connection [id: 0x76e2ccd1, /127.0.0.1:39418 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.269 [DEBUG - New I/O worker #476] (Negotiator.java:764) Authenticated connection [id: 0xfa767540, /127.0.0.1:39424 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.268 [DEBUG - New I/O worker #252] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.268 [DEBUG - New I/O worker #183] (Negotiator.java:764) Authenticated connection [id: 0x500d2e45, /127.0.0.1:39370 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.266 [DEBUG - New I/O worker #138] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.266 [DEBUG - New I/O worker #383] (Negotiator.java:764) Authenticated connection [id: 0xeb0c8e05, /127.0.0.1:39410 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.265 [DEBUG - New I/O worker #58] (Negotiator.java:764) Authenticated connection [id: 0xc422dbcf, /127.0.0.1:39268 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.263 [DEBUG - New I/O worker #220] (Negotiator.java:764) Authenticated connection [id: 0x641c3916, /127.0.0.1:37464 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.262 [DEBUG - New I/O worker #221] (Negotiator.java:764) Authenticated connection [id: 0xc15f8c2d, /127.0.0.1:37454 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.262 [DEBUG - New I/O worker #286] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.261 [DEBUG - New I/O worker #474] (Negotiator.java:764) Authenticated connection [id: 0x7b8c3b8b, /127.0.0.1:52794 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.261 [DEBUG - New I/O worker #145] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.261 [DEBUG - New I/O worker #470] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.260 [DEBUG - New I/O worker #302] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.328 [DEBUG - New I/O worker #418] (Negotiator.java:764) Authenticated connection [id: 0x32cf3069, /127.0.0.1:39478 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.260 [DEBUG - New I/O worker #274] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.260 [DEBUG - New I/O worker #492] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.259 [DEBUG - New I/O worker #494] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.258 [DEBUG - New I/O worker #448] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.258 [DEBUG - New I/O worker #397] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.258 [DEBUG - New I/O worker #375] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.257 [DEBUG - New I/O worker #346] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.256 [DEBUG - New I/O worker #128] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.256 [DEBUG - New I/O worker #323] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.252 [DEBUG - New I/O worker #328] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.252 [DEBUG - New I/O worker #333] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.251 [DEBUG - New I/O worker #388] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.250 [DEBUG - New I/O worker #118] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.247 [DEBUG - New I/O worker #90] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.247 [DEBUG - New I/O worker #70] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.243 [DEBUG - New I/O worker #100] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.241 [DEBUG - New I/O worker #108] (Negotiator.java:764) Authenticated connection [id: 0x72db7cc6, /127.0.0.1:37340 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.241 [DEBUG - New I/O worker #115] (Negotiator.java:764) Authenticated connection [id: 0x4abfe30d, /127.0.0.1:52776 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.240 [DEBUG - New I/O worker #121] (Negotiator.java:764) Authenticated connection [id: 0x965245b0, /127.0.0.1:52710 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.240 [DEBUG - New I/O worker #60] (Negotiator.java:764) Authenticated connection [id: 0x4277c25c, /127.0.0.1:39280 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.240 [DEBUG - New I/O worker #120] (Negotiator.java:764) Authenticated connection [id: 0x5aea0372, /127.0.0.1:52764 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.239 [DEBUG - New I/O worker #112] (Negotiator.java:764) Authenticated connection [id: 0x062ebe5d, /127.0.0.1:37310 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.239 [DEBUG - New I/O worker #306] (Negotiator.java:764) Authenticated connection [id: 0xffa7dbd6, /127.0.0.1:37412 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.238 [DEBUG - New I/O worker #133] (Negotiator.java:764) Authenticated connection [id: 0xcdd69dd0, /127.0.0.1:39226 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.237 [DEBUG - New I/O worker #265] (Negotiator.java:764) Authenticated connection [id: 0x928fb3ff, /127.0.0.1:52812 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.237 [DEBUG - New I/O worker #180] (Negotiator.java:764) Authenticated connection [id: 0x26ac1452, /127.0.0.1:52686 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.237 [DEBUG - New I/O worker #12] (Negotiator.java:764) Authenticated connection [id: 0xe904af3c, /127.0.0.1:37238 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.235 [DEBUG - New I/O worker #72] (Negotiator.java:764) Authenticated connection [id: 0xec695af0, /127.0.0.1:37418 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.235 [DEBUG - New I/O worker #43] (Negotiator.java:764) Authenticated connection [id: 0x3bada6db, /127.0.0.1:39256 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.233 [DEBUG - New I/O worker #114] (Negotiator.java:764) Authenticated connection [id: 0x19b862c0, /127.0.0.1:39286 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.231 [DEBUG - New I/O worker #69] (Negotiator.java:764) Authenticated connection [id: 0x105015b6, /127.0.0.1:39296 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.231 [DEBUG - New I/O worker #37] (Negotiator.java:764) Authenticated connection [id: 0xacd74def, /127.0.0.1:39244 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.230 [DEBUG - New I/O worker #10] (Negotiator.java:764) Authenticated connection [id: 0x049a314a, /127.0.0.1:37298 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.229 [DEBUG - New I/O worker #49] (Negotiator.java:764) Authenticated connection [id: 0xd1022900, /127.0.0.1:37274 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.229 [DEBUG - New I/O worker #124] (Negotiator.java:764) Authenticated connection [id: 0xc0ff97a6, /127.0.0.1:39266 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.228 [DEBUG - New I/O worker #301] (Negotiator.java:764) Authenticated connection [id: 0xd020d6fe, /127.0.0.1:39208 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.228 [DEBUG - New I/O worker #48] (Negotiator.java:764) Authenticated connection [id: 0x3dc468c6, /127.0.0.1:37334 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.228 [DEBUG - New I/O worker #119] (Negotiator.java:764) Authenticated connection [id: 0x628969a3, /127.0.0.1:39232 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.227 [DEBUG - New I/O worker #18] (Negotiator.java:764) Authenticated connection [id: 0xcbff8aea, /127.0.0.1:37288 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.226 [DEBUG - New I/O worker #102] (Negotiator.java:764) Authenticated connection [id: 0x1f18e105, /127.0.0.1:37378 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.225 [DEBUG - New I/O worker #105] (Negotiator.java:764) Authenticated connection [id: 0x14e3c8e3, /127.0.0.1:37240 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.223 [DEBUG - New I/O worker #129] (Negotiator.java:764) Authenticated connection [id: 0x3ef05f22, /127.0.0.1:39340 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.223 [DEBUG - New I/O worker #30] (Negotiator.java:764) Authenticated connection [id: 0x9793ff67, /127.0.0.1:37316 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.221 [DEBUG - New I/O worker #16] (Negotiator.java:764) Authenticated connection [id: 0xad6d0231, /127.0.0.1:52644 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.221 [DEBUG - New I/O worker #14] (Negotiator.java:764) Authenticated connection [id: 0x0c129717, /127.0.0.1:37226 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.220 [DEBUG - New I/O worker #53] (Negotiator.java:764) Authenticated connection [id: 0x75848a8b, /127.0.0.1:39304 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.219 [DEBUG - New I/O worker #127] (Negotiator.java:764) Authenticated connection [id: 0x43cc6d4d, /127.0.0.1:39310 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.218 [DEBUG - New I/O worker #47] (Negotiator.java:764) Authenticated connection [id: 0xf2508776, /127.0.0.1:37262 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.218 [DEBUG - New I/O worker #64] (Negotiator.java:764) Authenticated connection [id: 0x0cb81e28, /127.0.0.1:52730 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.215 [DEBUG - New I/O worker #420] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.214 [DEBUG - New I/O worker #79] (Negotiator.java:764) Authenticated connection [id: 0x48a169f9, /127.0.0.1:37318 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.213 [DEBUG - New I/O worker #304] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.213 [DEBUG - New I/O worker #25] (Negotiator.java:764) Authenticated connection [id: 0x27f78050, /127.0.0.1:52674 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.212 [DEBUG - New I/O worker #84] (Negotiator.java:764) Authenticated connection [id: 0x7154440b, /127.0.0.1:39250 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.066 [DEBUG - New I/O worker #143] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.363 [DEBUG - New I/O worker #350] (Negotiator.java:764) Authenticated connection [id: 0x6c77c1f7, /127.0.0.1:52874 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.387 [DEBUG - New I/O worker #93] (Negotiator.java:764) Authenticated connection [id: 0x7471b130, /127.0.0.1:39416 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.388 [DEBUG - New I/O worker #442] (Negotiator.java:764) Authenticated connection [id: 0xcadefec9, /127.0.0.1:39454 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.403 [DEBUG - New I/O worker #374] (Negotiator.java:764) Authenticated connection [id: 0x4b88e1aa, /127.0.0.1:39484 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.472 [DEBUG - New I/O worker #267] (Negotiator.java:764) Authenticated connection [id: 0x4971a4e5, /127.0.0.1:37612 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.479 [DEBUG - New I/O worker #475] (Negotiator.java:764) Authenticated connection [id: 0x01152b90, /127.0.0.1:37496 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.496 [DEBUG - New I/O worker #302] (Negotiator.java:764) Authenticated connection [id: 0x2dde52a8, /127.0.0.1:52898 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.504 [DEBUG - New I/O worker #270] (Negotiator.java:764) Authenticated connection [id: 0xff502aca, /127.0.0.1:37520 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.509 [DEBUG - New I/O worker #290] (Negotiator.java:764) Authenticated connection [id: 0x80202aba, /127.0.0.1:52884 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.511 [DEBUG - New I/O worker #142] (Negotiator.java:764) Authenticated connection [id: 0x229d9e13, /127.0.0.1:37538 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.517 [DEBUG - New I/O worker #138] (Negotiator.java:764) Authenticated connection [id: 0x2833c3a6, /127.0.0.1:37544 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.531 [DEBUG - New I/O worker #273] (Negotiator.java:764) Authenticated connection [id: 0xf6311a58, /127.0.0.1:37484 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.539 [DEBUG - New I/O worker #269] (Negotiator.java:764) Authenticated connection [id: 0x4c7a1ae1, /127.0.0.1:37528 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.541 [DEBUG - New I/O worker #293] (Negotiator.java:764) Authenticated connection [id: 0x8aa7209d, /127.0.0.1:52878 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.544 [DEBUG - New I/O worker #352] (Negotiator.java:764) Authenticated connection [id: 0xba87d4a8, /127.0.0.1:39504 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.545 [DEBUG - New I/O worker #416] (Negotiator.java:764) Authenticated connection [id: 0x41d30afd, /127.0.0.1:52964 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.547 [DEBUG - New I/O worker #252] (Negotiator.java:764) Authenticated connection [id: 0x891efe0b, /127.0.0.1:37552 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.548 [DEBUG - New I/O worker #128] (Negotiator.java:764) Authenticated connection [id: 0xf28b1856, /127.0.0.1:52950 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.548 [DEBUG - New I/O worker #304] (Negotiator.java:764) Authenticated connection [id: 0xd9739a21, /127.0.0.1:52918 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.551 [DEBUG - New I/O worker #90] (Negotiator.java:764) Authenticated connection [id: 0xc0c96f1f, /127.0.0.1:52928 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.552 [DEBUG - New I/O worker #81] (Negotiator.java:764) Authenticated connection [id: 0x7ed752fd, /127.0.0.1:37602 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.534 [DEBUG - New I/O worker #324] (Negotiator.java:764) Authenticated connection [id: 0xab8c9adf, /127.0.0.1:37490 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.556 [DEBUG - New I/O worker #260] (Negotiator.java:764) Authenticated connection [id: 0x649425fd, /127.0.0.1:37514 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.557 [DEBUG - New I/O worker #17] (Negotiator.java:764) Authenticated connection [id: 0xb163691c, /127.0.0.1:37376 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.559 [DEBUG - New I/O worker #141] (Negotiator.java:764) Authenticated connection [id: 0x7ee7f941, /127.0.0.1:37526 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.539 [DEBUG - New I/O worker #470] (Negotiator.java:764) Authenticated connection [id: 0x422b32ca, /127.0.0.1:37562 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.537 [DEBUG - New I/O worker #365] (Negotiator.java:764) Authenticated connection [id: 0x0b7f33c0, /127.0.0.1:39460 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.561 [DEBUG - New I/O worker #390] (Negotiator.java:764) Authenticated connection [id: 0xb533b1a0, /127.0.0.1:39490 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.537 [DEBUG - New I/O worker #143] (Negotiator.java:764) Authenticated connection [id: 0x1778a57e, /127.0.0.1:37502 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.536 [DEBUG - New I/O worker #492] (Negotiator.java:764) Authenticated connection [id: 0x240e05b7, /127.0.0.1:52908 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.534 [DEBUG - New I/O worker #323] (Negotiator.java:764) Authenticated connection [id: 0x905efa69, /127.0.0.1:52948 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.567 [DEBUG - New I/O worker #100] (Negotiator.java:764) Authenticated connection [id: 0x893f26f9, /127.0.0.1:37572 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.627 [DEBUG - New I/O worker #434] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.627 [DEBUG - New I/O worker #192] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.629 [DEBUG - New I/O worker #218] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.627 [DEBUG - New I/O worker #231] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.645 [DEBUG - New I/O worker #218] (Negotiator.java:764) Authenticated connection [id: 0x551a543c, /127.0.0.1:37588 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.646 [DEBUG - New I/O worker #182] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.648 [DEBUG - New I/O worker #246] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.650 [DEBUG - New I/O worker #257] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.650 [DEBUG - New I/O worker #414] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.653 [DEBUG - New I/O worker #349] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.654 [DEBUG - New I/O worker #179] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.656 [DEBUG - New I/O worker #434] (Negotiator.java:764) Authenticated connection [id: 0x29ab3cdd, /127.0.0.1:39588 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.653 [DEBUG - New I/O worker #412] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.655 [DEBUG - New I/O worker #297] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.655 [DEBUG - New I/O worker #110] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.655 [DEBUG - New I/O worker #247] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.663 [DEBUG - New I/O worker #266] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.654 [DEBUG - New I/O worker #264] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.665 [DEBUG - New I/O worker #184] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.668 [DEBUG - New I/O worker #373] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.678 [DEBUG - New I/O worker #414] (Negotiator.java:764) Authenticated connection [id: 0xb5e92e2a, /127.0.0.1:39606 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.691 [DEBUG - New I/O worker #373] (Negotiator.java:764) Authenticated connection [id: 0x75c5df94, /127.0.0.1:39600 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.710 [DEBUG - New I/O worker #179] (Negotiator.java:764) Authenticated connection [id: 0x50c325ad, /127.0.0.1:52980 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.712 [DEBUG - New I/O worker #184] (Negotiator.java:764) Authenticated connection [id: 0x871e7349, /127.0.0.1:39610 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.725 [DEBUG - New I/O worker #246] (Negotiator.java:764) Authenticated connection [id: 0x6aba12f9, /127.0.0.1:37600 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.726 [DEBUG - New I/O worker #257] (Negotiator.java:764) Authenticated connection [id: 0x3b4102cf, /127.0.0.1:37594 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.728 [DEBUG - Time-limited test] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:33.732 [DEBUG - pool-3-thread-78] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:33.736 [DEBUG - New I/O worker #481] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.738 [DEBUG - New I/O worker #483] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.738 [DEBUG - New I/O worker #486] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.736 [DEBUG - New I/O worker #360] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.740 [DEBUG - New I/O worker #478] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.739 [DEBUG - New I/O worker #597] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.747 [DEBUG - New I/O worker #412] (Negotiator.java:764) Authenticated connection [id: 0xc76902d0, /127.0.0.1:52976 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.747 [DEBUG - New I/O worker #517] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.750 [DEBUG - New I/O worker #485] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.753 [DEBUG - New I/O worker #291] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.755 [DEBUG - New I/O worker #305] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.755 [DEBUG - New I/O worker #271] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.757 [DEBUG - New I/O worker #466] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.768 [DEBUG - New I/O worker #396] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.774 [DEBUG - New I/O worker #466] (Negotiator.java:764) Authenticated connection [id: 0x8e1fb6db, /127.0.0.1:39642 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.774 [DEBUG - New I/O worker #351] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.775 [DEBUG - New I/O worker #593] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.781 [DEBUG - New I/O worker #486] (Negotiator.java:764) Authenticated connection [id: 0x2fe43951, /127.0.0.1:52992 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.795 [DEBUG - New I/O worker #483] (Negotiator.java:764) Authenticated connection [id: 0x29289c36, /127.0.0.1:39616 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.803 [DEBUG - New I/O worker #344] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.808 [DEBUG - New I/O worker #379] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.814 [DEBUG - New I/O worker #271] (Negotiator.java:764) Authenticated connection [id: 0x0ec984bf, /127.0.0.1:37666 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.817 [DEBUG - New I/O worker #481] (Negotiator.java:764) Authenticated connection [id: 0x82b537fa, /127.0.0.1:52986 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.820 [DEBUG - New I/O worker #297] (Negotiator.java:764) Authenticated connection [id: 0x47cb8ab4, /127.0.0.1:52934 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.825 [DEBUG - New I/O worker #379] (Negotiator.java:764) Authenticated connection [id: 0x3cf5465a, /127.0.0.1:39646 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.825 [DEBUG - pool-3-thread-51] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:33.827 [DEBUG - New I/O worker #631] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.835 [DEBUG - New I/O worker #351] (Negotiator.java:764) Authenticated connection [id: 0xb4f153f0, /127.0.0.1:53012 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.837 [DEBUG - New I/O worker #490] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.839 [DEBUG - New I/O worker #274] (Negotiator.java:764) Authenticated connection [id: 0xa5c3f83a, /127.0.0.1:37550 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.841 [DEBUG - New I/O worker #588] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.835 [DEBUG - New I/O worker #472] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.836 [DEBUG - New I/O worker #344] (Negotiator.java:764) Authenticated connection [id: 0xe5b1202e, /127.0.0.1:53016 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.836 [DEBUG - New I/O worker #272] (Negotiator.java:764) Authenticated connection [id: 0xb1343b9c, /127.0.0.1:37618 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.835 [DEBUG - New I/O worker #70] (Negotiator.java:764) Authenticated connection [id: 0x3e02b5f3, /127.0.0.1:37582 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.896 [DEBUG - New I/O worker #15] (Negotiator.java:764) Authenticated connection [id: 0x4d165f29, /127.0.0.1:37280 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.897 [DEBUG - New I/O worker #402] (Negotiator.java:764) Authenticated connection [id: 0xa03c2857, /127.0.0.1:52958 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.900 [DEBUG - New I/O worker #494] (Negotiator.java:764) Authenticated connection [id: 0x6aa48c0b, /127.0.0.1:39538 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.900 [DEBUG - New I/O worker #448] (Negotiator.java:764) Authenticated connection [id: 0x5c0d9754, /127.0.0.1:39530 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.900 [DEBUG - New I/O worker #15] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:33.901 [DEBUG - New I/O worker #15] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2410), Traces: [0ms] refreshing cache from master, [65ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [65ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [66ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1036ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1939ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2410ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:33.900 [DEBUG - New I/O worker #24] (Negotiator.java:764) Authenticated connection [id: 0x35c6a995, /127.0.0.1:39202 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.902 [DEBUG - New I/O worker #376] (Negotiator.java:764) Authenticated connection [id: 0x0f572d66, /127.0.0.1:39496 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.032 [DEBUG - New I/O worker #252] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.033 [DEBUG - New I/O worker #252] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1283), Traces: [0ms] refreshing cache from master, [334ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [334ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [335ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [800ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1185ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1241ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.031 [DEBUG - New I/O worker #342] (Negotiator.java:764) Authenticated connection [id: 0xf89cf81b, /127.0.0.1:39508 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.031 [DEBUG - New I/O worker #260] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.041 [DEBUG - New I/O worker #260] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1286), Traces: [0ms] refreshing cache from master, [167ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [168ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [169ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [283ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1141ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1097ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.029 [DEBUG - New I/O worker #420] (Negotiator.java:764) Authenticated connection [id: 0xc7d40d27, /127.0.0.1:39550 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.028 [DEBUG - New I/O worker #118] (Negotiator.java:764) Authenticated connection [id: 0x93a48cd3, /127.0.0.1:39558 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.027 [DEBUG - New I/O worker #586] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:34.051 [DEBUG - New I/O worker #588] (Negotiator.java:764) Authenticated connection [id: 0xd3859985, /127.0.0.1:39652 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.025 [DEBUG - New I/O worker #22] (Negotiator.java:764) Authenticated connection [id: 0xe576257d, /127.0.0.1:37358 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.024 [DEBUG - New I/O worker #407] (Negotiator.java:764) Authenticated connection [id: 0x63659d7b, /127.0.0.1:39514 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.054 [DEBUG - New I/O worker #631] (Negotiator.java:764) Authenticated connection [id: 0x42d94511, /127.0.0.1:39636 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.056 [DEBUG - New I/O worker #22] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.056 [DEBUG - New I/O worker #22] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2569), Traces: [0ms] refreshing cache from master, [280ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [280ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1469ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2568ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.024 [DEBUG - New I/O worker #482] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:34.023 [DEBUG - New I/O worker #286] (Negotiator.java:764) Authenticated connection [id: 0x3c68c6eb, /127.0.0.1:39520 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.019 [DEBUG - New I/O worker #360] (Negotiator.java:764) Authenticated connection [id: 0xc44e6c18, /127.0.0.1:37640 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.019 [DEBUG - New I/O worker #264] (Negotiator.java:764) Authenticated connection [id: 0x249b202c, /127.0.0.1:37624 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.019 [DEBUG - New I/O worker #274] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.106 [DEBUG - New I/O worker #274] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1353), Traces: [0ms] refreshing cache from master, [331ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [332ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [332ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1266ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.018 [DEBUG - New I/O worker #359] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:34.017 [DEBUG - New I/O worker #291] (Negotiator.java:764) Authenticated connection [id: 0x38f7eda2, /127.0.0.1:37652 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.017 [DEBUG - New I/O worker #271] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.126 [DEBUG - New I/O worker #271] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1347), Traces: [0ms] refreshing cache from master, [975ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [975ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [975ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [998ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1073ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1238ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.016 [DEBUG - New I/O worker #272] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.126 [DEBUG - New I/O worker #272] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1677), Traces: [0ms] refreshing cache from master, [845ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [846ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [847ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1453ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1567ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.015 [DEBUG - New I/O worker #660] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:34.015 [DEBUG - New I/O worker #493] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:34.014 [DEBUG - New I/O worker #70] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.140 [DEBUG - New I/O worker #70] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1677), Traces: [0ms] refreshing cache from master, [781ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [781ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [782ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1447ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1552ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1603ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.014 [DEBUG - New I/O worker #277] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:34.014 [DEBUG - New I/O worker #597] (Negotiator.java:764) Authenticated connection [id: 0x1227c062, /127.0.0.1:39622 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.012 [DEBUG - New I/O worker #478] (Negotiator.java:764) Authenticated connection [id: 0xdd2230d8, /127.0.0.1:37646 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.159 [DEBUG - New I/O worker #291] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.011 [DEBUG - New I/O worker #27] (Negotiator.java:764) Authenticated connection [id: 0x9cc4442d, /127.0.0.1:37400 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.009 [DEBUG - New I/O worker #485] (Negotiator.java:764) Authenticated connection [id: 0x9ba1d186, /127.0.0.1:37660 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.162 [DEBUG - New I/O worker #478] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.162 [DEBUG - New I/O worker #478] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1379), Traces: [0ms] refreshing cache from master, [952ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [952ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [953ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1004ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1380ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.005 [DEBUG - New I/O worker #517] (Negotiator.java:764) Authenticated connection [id: 0x58b350a9, /127.0.0.1:39630 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.003 [DEBUG - New I/O worker #305] (Negotiator.java:764) Authenticated connection [id: 0x7ef4cb77, /127.0.0.1:37654 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.000 [DEBUG - New I/O worker #396] (Negotiator.java:764) Authenticated connection [id: 0xf71df3d2, /127.0.0.1:53000 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.990 [DEBUG - New I/O worker #15] (Deferred.java:1330) callback=retry RPC@772154561 returned Deferred@708425177(state=PENDING, result=null, callback=(continuation of Deferred@1866149626 after retry RPC@772154561), errback=(continuation of Deferred@1866149626 after retry RPC@772154561)), so the following Deferred is getting paused: Deferred@1866149626(state=PAUSED, result=Deferred@708425177, callback=org.apache.kudu.client.AsyncKuduClient$6@41219122 -> wakeup thread pool-3-thread-4, errback=passthrough -> wakeup thread pool-3-thread-4) 00:27:34.170 [DEBUG - New I/O worker #15] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:33.999 [DEBUG - New I/O worker #375] (Negotiator.java:764) Authenticated connection [id: 0xbc91b376, /127.0.0.1:39532 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.994 [DEBUG - New I/O worker #79] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.189 [DEBUG - New I/O worker #79] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2693), Traces: [0ms] refreshing cache from master, [74ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [75ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [75ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [2499ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:33.981 [DEBUG - New I/O worker #92] (Negotiator.java:764) Authenticated connection [id: 0x047a96af, /127.0.0.1:37430 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.977 [DEBUG - New I/O worker #231] (Negotiator.java:764) Authenticated connection [id: 0xcd19616e, /127.0.0.1:39552 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.973 [DEBUG - New I/O worker #593] (Negotiator.java:764) Authenticated connection [id: 0x3f8b2908, /127.0.0.1:53006 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.970 [DEBUG - New I/O worker #192] (Negotiator.java:764) Authenticated connection [id: 0x09424154, /127.0.0.1:52920 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.969 [DEBUG - New I/O worker #35] (Negotiator.java:764) Authenticated connection [id: 0xb0f28e5f, /127.0.0.1:37428 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.968 [DEBUG - New I/O worker #246] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:33.968 [DEBUG - New I/O worker #109] (Negotiator.java:764) Authenticated connection [id: 0xfe4bd8e3, /127.0.0.1:52698 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.967 [DEBUG - New I/O worker #181] (Negotiator.java:764) Authenticated connection [id: 0x3d5d5f7c, /127.0.0.1:52740 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.967 [DEBUG - New I/O worker #528] (Negotiator.java:764) Authenticated connection [id: 0x44e0bbf3, /127.0.0.1:39466 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.967 [DEBUG - New I/O worker #103] (Negotiator.java:764) Authenticated connection [id: 0x14eb1ca1, /127.0.0.1:39364 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.966 [DEBUG - New I/O worker #126] (Negotiator.java:764) Authenticated connection [id: 0x73dbf401, /127.0.0.1:39376 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.966 [DEBUG - New I/O worker #182] (Negotiator.java:764) Authenticated connection [id: 0x70faaa8e, /127.0.0.1:39582 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.962 [DEBUG - New I/O worker #326] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.961 [DEBUG - New I/O worker #387] (Negotiator.java:764) Authenticated connection [id: 0xee6fc9c5, /127.0.0.1:52758 => /127.22.19.254:44849] using SASL/PLAIN 00:27:34.211 [DEBUG - New I/O worker #246] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1752), Traces: [0ms] refreshing cache from master, [798ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [799ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [799ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1112ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1514ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:33.960 [DEBUG - New I/O worker #39] (Negotiator.java:764) Authenticated connection [id: 0x87c03791, /127.0.0.1:52650 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.956 [DEBUG - New I/O worker #110] (Negotiator.java:764) Authenticated connection [id: 0xe7748c97, /127.0.0.1:37568 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.951 [DEBUG - New I/O worker #41] (Negotiator.java:764) Authenticated connection [id: 0x3bf53301, /127.0.0.1:39218 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.950 [DEBUG - New I/O worker #175] (Negotiator.java:764) Authenticated connection [id: 0x438bbd0d, /127.0.0.1:52752 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.949 [DEBUG - New I/O worker #193] (Negotiator.java:764) Authenticated connection [id: 0x06ecc412, /127.0.0.1:52728 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.948 [DEBUG - New I/O worker #71] (Negotiator.java:764) Authenticated connection [id: 0xfdcce0dc, /127.0.0.1:37256 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.948 [DEBUG - New I/O worker #247] (Negotiator.java:764) Authenticated connection [id: 0x8bbda5e2, /127.0.0.1:37570 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.947 [DEBUG - New I/O worker #266] (Negotiator.java:764) Authenticated connection [id: 0x3e6f1217, /127.0.0.1:37630 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.944 [DEBUG - New I/O worker #275] (Negotiator.java:764) Authenticated connection [id: 0x2a699c29, /127.0.0.1:37508 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.943 [DEBUG - New I/O worker #217] (Negotiator.java:764) Authenticated connection [id: 0x5bbc4600, /127.0.0.1:37424 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.941 [DEBUG - New I/O worker #349] (Negotiator.java:764) Authenticated connection [id: 0x58d17d19, /127.0.0.1:52970 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.936 [DEBUG - New I/O worker #425] (Negotiator.java:764) Authenticated connection [id: 0xe1e8b048, /127.0.0.1:39594 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.935 [DEBUG - New I/O worker #116] (Negotiator.java:764) Authenticated connection [id: 0x9e476070, /127.0.0.1:37394 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.934 [DEBUG - New I/O worker #306] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:33.934 [DEBUG - New I/O worker #50] (Negotiator.java:764) Authenticated connection [id: 0x53b6ebc4, /127.0.0.1:39360 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.228 [DEBUG - New I/O worker #306] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2151), Traces: [0ms] refreshing cache from master, [166ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [167ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [167ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [912ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1858ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [2145ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:33.933 [DEBUG - New I/O worker #74] (Negotiator.java:764) Authenticated connection [id: 0xf9b772e4, /127.0.0.1:52692 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.933 [DEBUG - New I/O worker #67] (Negotiator.java:764) Authenticated connection [id: 0x28ccab3a, /127.0.0.1:37352 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.933 [DEBUG - New I/O worker #292] (Negotiator.java:764) Authenticated connection [id: 0x4884f0b3, /127.0.0.1:37434 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.932 [DEBUG - New I/O worker #123] (Negotiator.java:764) Authenticated connection [id: 0x45602657, /127.0.0.1:39298 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.931 [DEBUG - New I/O worker #78] (Negotiator.java:764) Authenticated connection [id: 0x678fafe5, /127.0.0.1:37286 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.931 [DEBUG - New I/O worker #398] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:34.252 [DEBUG - New I/O worker #306] (Deferred.java:1330) callback=retry RPC@877261787 returned Deferred@813068682(state=PENDING, result=null, callback=(continuation of Deferred@1970600416 after retry RPC@877261787), errback=(continuation of Deferred@1970600416 after retry RPC@877261787)), so the following Deferred is getting paused: Deferred@1970600416(state=PAUSED, result=Deferred@813068682, callback=org.apache.kudu.client.AsyncKuduClient$6@3fe7647e -> wakeup thread pool-3-thread-70, errback=passthrough -> wakeup thread pool-3-thread-70) 00:27:33.931 [DEBUG - New I/O worker #137] (Negotiator.java:764) Authenticated connection [id: 0x05cc3d7b, /127.0.0.1:37370 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.931 [DEBUG - New I/O worker #17] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.275 [DEBUG - New I/O worker #17] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2314), Traces: [0ms] refreshing cache from master, [8ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [9ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [9ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1329ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1886ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [2277ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:33.931 [DEBUG - New I/O worker #241] (Negotiator.java:764) Authenticated connection [id: 0xc3956c9e, /127.0.0.1:39362 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.930 [DEBUG - New I/O worker #324] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.283 [DEBUG - New I/O worker #324] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1512), Traces: [0ms] refreshing cache from master, [108ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [108ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [108ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [234ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1074ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1451ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:33.930 [DEBUG - New I/O worker #356] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.929 [DEBUG - New I/O worker #388] (Negotiator.java:764) Authenticated connection [id: 0xec844b6f, /127.0.0.1:39570 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.929 [DEBUG - New I/O worker #23] (Negotiator.java:764) Authenticated connection [id: 0x6aede9e5, /127.0.0.1:52614 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.929 [DEBUG - New I/O worker #141] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.305 [DEBUG - New I/O worker #141] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1559), Traces: [0ms] refreshing cache from master, [300ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [322ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [322ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1098ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1107ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1311ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.305 [DEBUG - New I/O worker #398] (Negotiator.java:764) Authenticated connection [id: 0x61660cff, /127.0.0.1:39676 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.929 [DEBUG - New I/O worker #91] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.315 [DEBUG - New I/O worker #91] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2821), Traces: [0ms] refreshing cache from master, [26ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [28ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [29ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [950ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1916ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:33.929 [DEBUG - New I/O worker #10] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.318 [DEBUG - New I/O worker #10] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2822), Traces: [0ms] refreshing cache from master, [71ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [72ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [72ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1160ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1939ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2331ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:33.928 [DEBUG - New I/O worker #218] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.322 [DEBUG - New I/O worker #218] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1860), Traces: [0ms] refreshing cache from master, [783ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [783ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [784ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1187ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1370ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:33.928 [DEBUG - New I/O worker #111] (Negotiator.java:764) Authenticated connection [id: 0x5055107b, /127.0.0.1:39328 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.927 [DEBUG - New I/O worker #100] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:33.927 [DEBUG - New I/O worker #30] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.330 [DEBUG - New I/O worker #100] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1854), Traces: [0ms] refreshing cache from master, [734ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [734ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [735ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1348ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1739ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1745ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:33.927 [DEBUG - New I/O worker #470] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.342 [DEBUG - New I/O worker #470] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1595), Traces: [0ms] refreshing cache from master, [341ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [341ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [341ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1092ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1165ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1183ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.343 [DEBUG - New I/O worker #470] (Deferred.java:1330) callback=retry RPC@1944597229 returned Deferred@2010622024(state=PENDING, result=null, callback=(continuation of Deferred@852834518 after retry RPC@1944597229), errback=(continuation of Deferred@852834518 after retry RPC@1944597229)), so the following Deferred is getting paused: Deferred@852834518(state=PAUSED, result=Deferred@2010622024, callback=org.apache.kudu.client.AsyncKuduClient$6@31e01bb7 -> wakeup thread pool-3-thread-79, errback=passthrough -> wakeup thread pool-3-thread-79) 00:27:33.926 [DEBUG - New I/O worker #143] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:33.925 [DEBUG - New I/O worker #185] (Negotiator.java:764) Authenticated connection [id: 0xb174f38c, /127.0.0.1:39316 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.925 [DEBUG - New I/O worker #18] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:33.925 [DEBUG - New I/O worker #47] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.350 [DEBUG - New I/O worker #18] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2853), Traces: [0ms] refreshing cache from master, [68ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [68ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [68ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1102ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2054ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2338ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.350 [DEBUG - New I/O worker #47] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2858), Traces: [0ms] refreshing cache from master, [54ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [54ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [55ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1021ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2342ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:33.925 [DEBUG - New I/O worker #273] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:33.925 [DEBUG - New I/O worker #81] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.365 [DEBUG - New I/O worker #273] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1592), Traces: [0ms] refreshing cache from master, [103ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [103ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [104ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [294ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [779ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1062ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.365 [DEBUG - New I/O worker #81] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1909), Traces: [0ms] refreshing cache from master, [797ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [798ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [798ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1112ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1452ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1766ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.365 [DEBUG - New I/O worker #47] (Deferred.java:1330) callback=retry RPC@476940799 returned Deferred@408034986(state=PENDING, result=null, callback=(continuation of Deferred@1565557700 after retry RPC@476940799), errback=(continuation of Deferred@1565557700 after retry RPC@476940799)), so the following Deferred is getting paused: Deferred@1565557700(state=PAUSED, result=Deferred@408034986, callback=org.apache.kudu.client.AsyncKuduClient$6@1c153911 -> wakeup thread pool-3-thread-16, errback=passthrough -> wakeup thread pool-3-thread-16) 00:27:34.365 [DEBUG - New I/O worker #81] (Deferred.java:1330) callback=retry RPC@665005893 returned Deferred@597471081(state=PENDING, result=null, callback=(continuation of Deferred@1721640318 after retry RPC@665005893), errback=(continuation of Deferred@1721640318 after retry RPC@665005893)), so the following Deferred is getting paused: Deferred@1721640318(state=PAUSED, result=Deferred@597471081, callback=org.apache.kudu.client.AsyncKuduClient$6@695561b8 -> wakeup thread pool-3-thread-19, errback=passthrough -> wakeup thread pool-3-thread-19) 00:27:33.925 [DEBUG - New I/O worker #257] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.366 [DEBUG - New I/O worker #47] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:33.925 [DEBUG - New I/O worker #269] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.366 [DEBUG - New I/O worker #269] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1618), Traces: [0ms] refreshing cache from master, [300ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [300ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [300ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [733ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [814ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1074ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.367 [DEBUG - New I/O worker #269] (Deferred.java:1330) callback=retry RPC@1680045103 returned Deferred@1612512371(state=PENDING, result=null, callback=(continuation of Deferred@622755348 after retry RPC@1680045103), errback=(continuation of Deferred@622755348 after retry RPC@1680045103)), so the following Deferred is getting paused: Deferred@622755348(state=PAUSED, result=Deferred@1612512371, callback=org.apache.kudu.client.AsyncKuduClient$6@6b8ca93b -> wakeup thread pool-3-thread-59, errback=passthrough -> wakeup thread pool-3-thread-59) 00:27:33.924 [DEBUG - New I/O worker #138] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.367 [DEBUG - New I/O worker #138] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1617), Traces: [0ms] refreshing cache from master, [329ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [329ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [330ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [812ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1352ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.368 [DEBUG - New I/O worker #138] (Deferred.java:1330) callback=retry RPC@1672323490 returned Deferred@1737632015(state=PENDING, result=null, callback=(continuation of Deferred@579906457 after retry RPC@1672323490), errback=(continuation of Deferred@579906457 after retry RPC@1672323490)), so the following Deferred is getting paused: Deferred@579906457(state=PAUSED, result=Deferred@1737632015, callback=org.apache.kudu.client.AsyncKuduClient$6@1d566c0e -> wakeup thread pool-3-thread-38, errback=passthrough -> wakeup thread pool-3-thread-38) 00:27:33.924 [DEBUG - New I/O worker #219] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:33.924 [DEBUG - New I/O worker #142] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.369 [DEBUG - New I/O worker #219] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1596), Traces: [0ms] refreshing cache from master, [67ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [67ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [68ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [224ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [551ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [607ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.369 [DEBUG - New I/O worker #142] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1614), Traces: [0ms] refreshing cache from master, [320ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [320ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [321ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [759ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [794ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1314ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.369 [DEBUG - New I/O worker #219] (Deferred.java:1330) callback=retry RPC@1604599794 returned Deferred@1536925918(state=PENDING, result=null, callback=(continuation of Deferred@513357257 after retry RPC@1604599794), errback=(continuation of Deferred@513357257 after retry RPC@1604599794)), so the following Deferred is getting paused: Deferred@513357257(state=PAUSED, result=Deferred@1536925918, callback=org.apache.kudu.client.AsyncKuduClient$6@51e8a30f -> wakeup thread pool-3-thread-49, errback=passthrough -> wakeup thread pool-3-thread-49) 00:27:34.369 [DEBUG - New I/O worker #142] (Deferred.java:1330) callback=retry RPC@2042616621 returned Deferred@2105561282(state=PENDING, result=null, callback=(continuation of Deferred@948097302 after retry RPC@2042616621), errback=(continuation of Deferred@948097302 after retry RPC@2042616621)), so the following Deferred is getting paused: Deferred@948097302(state=PAUSED, result=Deferred@2105561282, callback=org.apache.kudu.client.AsyncKuduClient$6@e11d81f -> wakeup thread pool-3-thread-41, errback=passthrough -> wakeup thread pool-3-thread-41) 00:27:33.924 [DEBUG - New I/O worker #475] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.370 [DEBUG - New I/O worker #81] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.379 [DEBUG - New I/O worker #138] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:33.924 [DEBUG - New I/O worker #112] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.379 [DEBUG - New I/O worker #112] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2890), Traces: [0ms] refreshing cache from master, [80ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [81ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [81ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1046ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1911ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1989ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.380 [DEBUG - New I/O worker #112] (Deferred.java:1330) callback=retry RPC@1219508668 returned Deferred@1284486258(state=PENDING, result=null, callback=(continuation of Deferred@160248711 after retry RPC@1219508668), errback=(continuation of Deferred@160248711 after retry RPC@1219508668)), so the following Deferred is getting paused: Deferred@160248711(state=PAUSED, result=Deferred@1284486258, callback=org.apache.kudu.client.AsyncKuduClient$6@717a727b -> wakeup thread pool-3-thread-30, errback=passthrough -> wakeup thread pool-3-thread-30) 00:27:33.924 [DEBUG - New I/O worker #105] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:33.923 [DEBUG - New I/O worker #49] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.380 [DEBUG - New I/O worker #105] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2886), Traces: [0ms] refreshing cache from master, [25ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [25ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [25ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1017ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1955ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.381 [DEBUG - New I/O worker #49] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2889), Traces: [0ms] refreshing cache from master, [59ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [60ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [60ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1044ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1955ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [2064ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.381 [DEBUG - New I/O worker #105] (Deferred.java:1330) callback=retry RPC@1108873008 returned Deferred@1176994731(state=PENDING, result=null, callback=(continuation of Deferred@52757771 after retry RPC@1108873008), errback=(continuation of Deferred@52757771 after retry RPC@1108873008)), so the following Deferred is getting paused: Deferred@52757771(state=PAUSED, result=Deferred@1176994731, callback=org.apache.kudu.client.AsyncKuduClient$6@308fac84 -> wakeup thread pool-3-thread-33, errback=passthrough -> wakeup thread pool-3-thread-33) 00:27:34.381 [DEBUG - New I/O worker #49] (Deferred.java:1330) callback=retry RPC@1566940265 returned Deferred@1499071261(state=PENDING, result=null, callback=(continuation of Deferred@475567698 after retry RPC@1566940265), errback=(continuation of Deferred@475567698 after retry RPC@1566940265)), so the following Deferred is getting paused: Deferred@475567698(state=PAUSED, result=Deferred@1499071261, callback=org.apache.kudu.client.AsyncKuduClient$6@5cf964b4 -> wakeup thread pool-3-thread-18, errback=passthrough -> wakeup thread pool-3-thread-18) 00:27:33.923 [DEBUG - New I/O worker #108] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.382 [DEBUG - New I/O worker #142] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:33.923 [DEBUG - New I/O worker #102] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.382 [DEBUG - New I/O worker #102] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2420), Traces: [0ms] refreshing cache from master, [6ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [7ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [7ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1474ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [2263ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2316ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:33.922 [DEBUG - New I/O worker #268] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.383 [DEBUG - New I/O worker #268] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1610), Traces: [0ms] refreshing cache from master, [48ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [49ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [49ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [125ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [614ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1055ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.384 [DEBUG - New I/O worker #268] (Deferred.java:1330) callback=retry RPC@2105888210 returned Deferred@2042287802(state=PENDING, result=null, callback=(continuation of Deferred@1018709993 after retry RPC@2105888210), errback=(continuation of Deferred@1018709993 after retry RPC@2105888210)), so the following Deferred is getting paused: Deferred@1018709993(state=PAUSED, result=Deferred@2042287802, callback=org.apache.kudu.client.AsyncKuduClient$6@5a721159 -> wakeup thread pool-3-thread-58, errback=passthrough -> wakeup thread pool-3-thread-58) 00:27:33.922 [DEBUG - New I/O worker #86] (Negotiator.java:764) Authenticated connection [id: 0x0776cf90, /127.0.0.1:52666 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.922 [DEBUG - New I/O worker #72] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.385 [DEBUG - New I/O worker #102] (Deferred.java:1330) callback=retry RPC@1376024861 returned Deferred@1446770565(state=PENDING, result=null, callback=(continuation of Deferred@322532134 after retry RPC@1376024861), errback=(continuation of Deferred@322532134 after retry RPC@1376024861)), so the following Deferred is getting paused: Deferred@322532134(state=PAUSED, result=Deferred@1446770565, callback=org.apache.kudu.client.AsyncKuduClient$6@53dd6180 -> wakeup thread pool-3-thread-34, errback=passthrough -> wakeup thread pool-3-thread-34) 00:27:33.921 [DEBUG - New I/O worker #253] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.919 [DEBUG - New I/O worker #139] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:34.386 [DEBUG - New I/O worker #49] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:33.918 [DEBUG - New I/O worker #299] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.918 [DEBUG - New I/O worker #262] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:33.917 [DEBUG - New I/O worker #75] (Negotiator.java:764) Authenticated connection [id: 0xe3b2b8e4, /127.0.0.1:52608 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.917 [DEBUG - New I/O worker #56] (Negotiator.java:764) Authenticated connection [id: 0x26b3bd88, /127.0.0.1:39334 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.916 [DEBUG - New I/O worker #12] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.391 [DEBUG - New I/O worker #268] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.392 [DEBUG - New I/O worker #12] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2895), Traces: [0ms] refreshing cache from master, [23ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [23ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [23ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1090ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1906ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [2737ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:33.915 [DEBUG - New I/O worker #117] (Negotiator.java:764) Authenticated connection [id: 0xfdc9a086, /127.0.0.1:52596 => /127.22.19.254:44849] using SASL/PLAIN 00:27:34.392 [DEBUG - New I/O worker #12] (Deferred.java:1330) callback=retry RPC@1369234630 returned Deferred@1436766765(state=PENDING, result=null, callback=(continuation of Deferred@279041789 after retry RPC@1369234630), errback=(continuation of Deferred@279041789 after retry RPC@1369234630)), so the following Deferred is getting paused: Deferred@279041789(state=PAUSED, result=Deferred@1436766765, callback=org.apache.kudu.client.AsyncKuduClient$6@16369465 -> wakeup thread pool-3-thread-2, errback=passthrough -> wakeup thread pool-3-thread-2) 00:27:33.915 [DEBUG - New I/O worker #14] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.393 [DEBUG - New I/O worker #14] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2973), Traces: [0ms] refreshing cache from master, [13ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [14ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [14ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1090ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1981ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [2516ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.393 [DEBUG - New I/O worker #14] (Deferred.java:1330) callback=retry RPC@40038457 returned Deferred@106783917(state=PENDING, result=null, callback=(continuation of Deferred@1130363394 after retry RPC@40038457), errback=(continuation of Deferred@1130363394 after retry RPC@40038457)), so the following Deferred is getting paused: Deferred@1130363394(state=PAUSED, result=Deferred@106783917, callback=org.apache.kudu.client.AsyncKuduClient$6@67a0de53 -> wakeup thread pool-3-thread-5, errback=passthrough -> wakeup thread pool-3-thread-5) 00:27:33.915 [DEBUG - New I/O worker #144] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:33.914 [DEBUG - New I/O worker #288] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:33.913 [DEBUG - New I/O worker #333] (Negotiator.java:764) Authenticated connection [id: 0x983b8732, /127.0.0.1:39564 => /127.22.19.253:37879] using SASL/PLAIN 00:27:33.912 [DEBUG - New I/O worker #220] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.418 [DEBUG - New I/O worker #220] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1646), Traces: [0ms] refreshing cache from master, [51ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [51ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [51ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [109ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [629ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [629ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:34.418 [DEBUG - New I/O worker #220] (Deferred.java:1330) callback=retry RPC@1688326954 returned Deferred@1620990822(state=PENDING, result=null, callback=(continuation of Deferred@631036177 after retry RPC@1688326954), errback=(continuation of Deferred@631036177 after retry RPC@1688326954)), so the following Deferred is getting paused: Deferred@631036177(state=PAUSED, result=Deferred@1620990822, callback=org.apache.kudu.client.AsyncKuduClient$6@7ea1bb34 -> wakeup thread pool-3-thread-47, errback=passthrough -> wakeup thread pool-3-thread-47) 00:27:33.912 [DEBUG - New I/O worker #145] (Negotiator.java:764) Authenticated connection [id: 0xa086f2e9, /127.0.0.1:52890 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.912 [DEBUG - New I/O worker #48] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.419 [DEBUG - New I/O worker #48] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2929), Traces: [0ms] refreshing cache from master, [82ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [82ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [82ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1025ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2050ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2337ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.420 [DEBUG - New I/O worker #48] (Deferred.java:1330) callback=retry RPC@1394985551 returned Deferred@1461343475(state=PENDING, result=null, callback=(continuation of Deferred@303612020 after retry RPC@1394985551), errback=(continuation of Deferred@303612020 after retry RPC@1394985551)), so the following Deferred is getting paused: Deferred@303612020(state=PAUSED, result=Deferred@1461343475, callback=org.apache.kudu.client.AsyncKuduClient$6@5e4aeb93 -> wakeup thread pool-3-thread-15, errback=passthrough -> wakeup thread pool-3-thread-15) 00:27:33.911 [DEBUG - New I/O worker #221] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.420 [DEBUG - New I/O worker #221] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1647), Traces: [0ms] refreshing cache from master, [48ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [48ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [48ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [264ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [627ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [627ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.421 [DEBUG - New I/O worker #221] (Deferred.java:1330) callback=retry RPC@1947815177 returned Deferred@1881589586(state=PENDING, result=null, callback=(continuation of Deferred@891569970 after retry RPC@1947815177), errback=(continuation of Deferred@891569970 after retry RPC@1947815177)), so the following Deferred is getting paused: Deferred@891569970(state=PAUSED, result=Deferred@1881589586, callback=org.apache.kudu.client.AsyncKuduClient$6@3a944208 -> wakeup thread pool-3-thread-45, errback=passthrough -> wakeup thread pool-3-thread-45) 00:27:33.910 [DEBUG - New I/O worker #322] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.422 [DEBUG - New I/O worker #322] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1650), Traces: [0ms] refreshing cache from master, [91ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [91ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [91ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [225ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [563ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1126ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.422 [DEBUG - New I/O worker #322] (Deferred.java:1330) callback=retry RPC@1127863832 returned Deferred@104550427(state=PENDING, result=null, callback=(continuation of Deferred@1121290057 after retry RPC@1127863832), errback=(continuation of Deferred@1121290057 after retry RPC@1127863832)), so the following Deferred is getting paused: Deferred@1121290057(state=PAUSED, result=Deferred@104550427, callback=org.apache.kudu.client.AsyncKuduClient$6@37771269 -> wakeup thread pool-3-thread-73, errback=passthrough -> wakeup thread pool-3-thread-73) 00:27:33.910 [DEBUG - New I/O worker #362] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.423 [DEBUG - New I/O worker #362] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1649), Traces: [0ms] refreshing cache from master, [27ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [27ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [28ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [272ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [617ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [662ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:33.909 [DEBUG - New I/O worker #106] (Negotiator.java:764) Authenticated connection [id: 0x156b4cd2, /127.0.0.1:52588 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.909 [DEBUG - New I/O worker #267] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.424 [DEBUG - New I/O worker #267] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1976), Traces: [0ms] refreshing cache from master, [811ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [812ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [824ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1214ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1378ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1456ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:34.425 [DEBUG - New I/O worker #267] (Deferred.java:1330) callback=retry RPC@1582589806 returned Deferred@1517021308(state=PENDING, result=null, callback=(continuation of Deferred@527002965 after retry RPC@1582589806), errback=(continuation of Deferred@527002965 after retry RPC@1582589806)), so the following Deferred is getting paused: Deferred@527002965(state=PAUSED, result=Deferred@1517021308, callback=org.apache.kudu.client.AsyncKuduClient$6@53ae0657 -> wakeup thread pool-3-thread-54, errback=passthrough -> wakeup thread pool-3-thread-54) 00:27:34.433 [DEBUG - New I/O worker #267] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:33.908 [DEBUG - New I/O worker #134] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:33.908 [DEBUG - New I/O worker #28] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.433 [DEBUG - New I/O worker #134] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2345), Traces: [0ms] refreshing cache from master, [143ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [144ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [144ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [730ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1453ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2139ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:33.908 [DEBUG - New I/O worker #270] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.434 [DEBUG - New I/O worker #362] (Deferred.java:1330) callback=retry RPC@2046544209 returned Deferred@1176232543(state=PENDING, result=null, callback=(continuation of Deferred@1212724325 after retry RPC@2046544209), errback=(continuation of Deferred@1212724325 after retry RPC@2046544209)), so the following Deferred is getting paused: Deferred@1212724325(state=PAUSED, result=Deferred@1176232543, callback=org.apache.kudu.client.AsyncKuduClient$6@3e42c1c6 -> wakeup thread pool-3-thread-76, errback=passthrough -> wakeup thread pool-3-thread-76) 00:27:33.908 [DEBUG - New I/O worker #46] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.434 [DEBUG - New I/O worker #46] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2947), Traces: [0ms] refreshing cache from master, [158ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [234ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [234ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1077ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1808ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2745ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:34.435 [DEBUG - New I/O worker #46] (Deferred.java:1330) callback=retry RPC@512680560 returned Deferred@1877692688(state=PENDING, result=null, callback=(continuation of Deferred@802823446 after retry RPC@512680560), errback=(continuation of Deferred@802823446 after retry RPC@512680560)), so the following Deferred is getting paused: Deferred@802823446(state=PAUSED, result=Deferred@1877692688, callback=org.apache.kudu.client.AsyncKuduClient$6@1755f0c7 -> wakeup thread pool-3-thread-17, errback=passthrough -> wakeup thread pool-3-thread-17) 00:27:33.907 [DEBUG - New I/O worker #19] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.436 [DEBUG - New I/O worker #19] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2947), Traces: [0ms] refreshing cache from master, [79ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [79ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [79ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1052ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1911ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2731ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:34.437 [DEBUG - New I/O worker #19] (Deferred.java:1330) callback=retry RPC@1783839229 returned Deferred@793885395(state=PENDING, result=null, callback=(continuation of Deferred@366262926 after retry RPC@1783839229), errback=(continuation of Deferred@366262926 after retry RPC@1783839229)), so the following Deferred is getting paused: Deferred@366262926(state=PAUSED, result=Deferred@793885395, callback=org.apache.kudu.client.AsyncKuduClient$6@1e34b32c -> wakeup thread pool-3-thread-9, errback=passthrough -> wakeup thread pool-3-thread-9) 00:27:33.907 [DEBUG - New I/O worker #11] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:33.907 [DEBUG - New I/O worker #13] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.437 [DEBUG - New I/O worker #11] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2945), Traces: [0ms] refreshing cache from master, [56ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [56ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [56ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1046ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [2331ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2826ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:34.438 [DEBUG - New I/O worker #13] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2943), Traces: [0ms] refreshing cache from master, [77ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [77ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [77ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [921ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1875ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2058ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.438 [DEBUG - New I/O worker #11] (Deferred.java:1330) callback=retry RPC@209272544 returned Deferred@138849207(state=PENDING, result=null, callback=(continuation of Deferred@1296315611 after retry RPC@209272544), errback=(continuation of Deferred@1296315611 after retry RPC@209272544)), so the following Deferred is getting paused: Deferred@1296315611(state=PAUSED, result=Deferred@138849207, callback=org.apache.kudu.client.AsyncKuduClient$6@39086d4c -> wakeup thread pool-3-thread-3, errback=passthrough -> wakeup thread pool-3-thread-3) 00:27:34.438 [DEBUG - New I/O worker #13] (Deferred.java:1330) callback=retry RPC@418612055 returned Deferred@483132561(state=PENDING, result=null, callback=(continuation of Deferred@1506707820 after retry RPC@418612055), errback=(continuation of Deferred@1506707820 after retry RPC@418612055)), so the following Deferred is getting paused: Deferred@1506707820(state=PAUSED, result=Deferred@483132561, callback=org.apache.kudu.client.AsyncKuduClient$6@6cb0aed3 -> wakeup thread pool-3-thread-6, errback=passthrough -> wakeup thread pool-3-thread-6) 00:27:33.852 [DEBUG - New I/O worker #76] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.439 [DEBUG - New I/O worker #76] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2715), Traces: [0ms] refreshing cache from master, [178ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [178ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [179ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [412ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1664ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1703ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:34.439 [DEBUG - New I/O worker #76] (Deferred.java:1330) callback=retry RPC@1905820429 returned Deferred@1973885414(state=PENDING, result=null, callback=(continuation of Deferred@816154934 after retry RPC@1905820429), errback=(continuation of Deferred@816154934 after retry RPC@1905820429)), so the following Deferred is getting paused: Deferred@816154934(state=PAUSED, result=Deferred@1973885414, callback=org.apache.kudu.client.AsyncKuduClient$6@34ea9676 -> wakeup thread pool-3-thread-22, errback=passthrough -> wakeup thread pool-3-thread-22) 00:27:33.906 [DEBUG - New I/O worker #248] (Negotiator.java:764) Authenticated connection [id: 0xdcde8a3f, /127.0.0.1:37232 => /127.22.19.252:32833] using SASL/PLAIN 00:27:33.905 [DEBUG - New I/O worker #328] (Negotiator.java:764) Authenticated connection [id: 0xcd7ae606, /127.0.0.1:52940 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.904 [DEBUG - New I/O worker #346] (Negotiator.java:764) Authenticated connection [id: 0xf6b55d09, /127.0.0.1:52900 => /127.22.19.254:44849] using SASL/PLAIN 00:27:33.904 [DEBUG - New I/O worker #397] (Negotiator.java:764) Authenticated connection [id: 0xcee47e9f, /127.0.0.1:39580 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.434 [DEBUG - New I/O worker #270] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1679), Traces: [0ms] refreshing cache from master, [170ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [170ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [170ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [552ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1070ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1284ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.442 [DEBUG - New I/O worker #270] (Deferred.java:1330) callback=retry RPC@1333967920 returned Deferred@1270685976(state=PENDING, result=null, callback=(continuation of Deferred@247445003 after retry RPC@1333967920), errback=(continuation of Deferred@247445003 after retry RPC@1333967920)), so the following Deferred is getting paused: Deferred@247445003(state=PAUSED, result=Deferred@1270685976, callback=org.apache.kudu.client.AsyncKuduClient$6@65c2bbbe -> wakeup thread pool-3-thread-53, errback=passthrough -> wakeup thread pool-3-thread-53) 00:27:34.434 [DEBUG - New I/O worker #134] (Deferred.java:1330) callback=retry RPC@1143578513 returned Deferred@1075188684(state=PENDING, result=null, callback=(continuation of Deferred@85235114 after retry RPC@1143578513), errback=(continuation of Deferred@85235114 after retry RPC@1143578513)), so the following Deferred is getting paused: Deferred@85235114(state=PAUSED, result=Deferred@1075188684, callback=org.apache.kudu.client.AsyncKuduClient$6@3c86262c -> wakeup thread pool-3-thread-40, errback=passthrough -> wakeup thread pool-3-thread-40) 00:27:34.433 [DEBUG - New I/O worker #28] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2470), Traces: [0ms] refreshing cache from master, [5ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [6ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [6ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [666ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1473ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2260ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.443 [DEBUG - New I/O worker #28] (Deferred.java:1330) callback=retry RPC@669308632 returned Deferred@601573116(state=PENDING, result=null, callback=(continuation of Deferred@1725553891 after retry RPC@669308632), errback=(continuation of Deferred@1725553891 after retry RPC@669308632)), so the following Deferred is getting paused: Deferred@1725553891(state=PAUSED, result=Deferred@601573116, callback=org.apache.kudu.client.AsyncKuduClient$6@56a6c2f9 -> wakeup thread pool-3-thread-14, errback=passthrough -> wakeup thread pool-3-thread-14) 00:27:34.431 [DEBUG - New I/O worker #48] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.431 [DEBUG - New I/O worker #322] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.431 [DEBUG - New I/O worker #221] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.412 [DEBUG - New I/O worker #102] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.407 [DEBUG - New I/O worker #14] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.406 [DEBUG - New I/O worker #12] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.391 [DEBUG - New I/O worker #219] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.385 [DEBUG - New I/O worker #72] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2387), Traces: [0ms] refreshing cache from master, [252ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [253ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [253ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [822ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1405ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1440ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:34.448 [DEBUG - New I/O worker #72] (Deferred.java:1330) callback=retry RPC@1641270638 returned Deferred@1709987212(state=PENDING, result=null, callback=(continuation of Deferred@552519509 after retry RPC@1641270638), errback=(continuation of Deferred@552519509 after retry RPC@1641270638)), so the following Deferred is getting paused: Deferred@552519509(state=PAUSED, result=Deferred@1709987212, callback=org.apache.kudu.client.AsyncKuduClient$6@6ec3ee6a -> wakeup thread pool-3-thread-27, errback=passthrough -> wakeup thread pool-3-thread-27) 00:27:34.384 [DEBUG - New I/O worker #105] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.383 [DEBUG - New I/O worker #47] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.449 [DEBUG - New I/O worker #47] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.450 [DEBUG - New I/O worker #47] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.450 [DEBUG - New I/O worker #47] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.382 [DEBUG - New I/O worker #112] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.382 [DEBUG - New I/O worker #108] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2890), Traces: [0ms] refreshing cache from master, [220ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1909ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1947ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [2874ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.451 [DEBUG - New I/O worker #108] (Deferred.java:1330) callback=retry RPC@1367176368 returned Deferred@1430452807(state=PENDING, result=null, callback=(continuation of Deferred@272662155 after retry RPC@1367176368), errback=(continuation of Deferred@272662155 after retry RPC@1367176368)), so the following Deferred is getting paused: Deferred@272662155(state=PAUSED, result=Deferred@1430452807, callback=org.apache.kudu.client.AsyncKuduClient$6@1b78669d -> wakeup thread pool-3-thread-32, errback=passthrough -> wakeup thread pool-3-thread-32) 00:27:34.379 [DEBUG - New I/O worker #269] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.370 [DEBUG - New I/O worker #475] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1600), Traces: [0ms] refreshing cache from master, [112ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [112ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [113ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [173ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [616ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [738ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.452 [DEBUG - New I/O worker #475] (Deferred.java:1330) callback=retry RPC@1131893430 returned Deferred@1195953664(state=PENDING, result=null, callback=(continuation of Deferred@38426765 after retry RPC@1131893430), errback=(continuation of Deferred@38426765 after retry RPC@1131893430)), so the following Deferred is getting paused: Deferred@38426765(state=PAUSED, result=Deferred@1195953664, callback=org.apache.kudu.client.AsyncKuduClient$6@4a19f691 -> wakeup thread pool-3-thread-77, errback=passthrough -> wakeup thread pool-3-thread-77) 00:27:34.366 [DEBUG - New I/O worker #257] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1900), Traces: [0ms] refreshing cache from master, [782ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [782ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [782ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1265ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1853ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.453 [DEBUG - New I/O worker #257] (Deferred.java:1330) callback=retry RPC@240764345 returned Deferred@174477482(state=PENDING, result=null, callback=(continuation of Deferred@1332006786 after retry RPC@240764345), errback=(continuation of Deferred@1332006786 after retry RPC@240764345)), so the following Deferred is getting paused: Deferred@1332006786(state=PAUSED, result=Deferred@174477482, callback=org.apache.kudu.client.AsyncKuduClient$6@3ed57d5f -> wakeup thread pool-3-thread-64, errback=passthrough -> wakeup thread pool-3-thread-64) 00:27:34.365 [DEBUG - New I/O worker #273] (Deferred.java:1330) callback=retry RPC@1894202883 returned Deferred@1960355015(state=PENDING, result=null, callback=(continuation of Deferred@836383800 after retry RPC@1894202883), errback=(continuation of Deferred@836383800 after retry RPC@1894202883)), so the following Deferred is getting paused: Deferred@836383800(state=PAUSED, result=Deferred@1960355015, callback=org.apache.kudu.client.AsyncKuduClient$6@49ff08c3 -> wakeup thread pool-3-thread-57, errback=passthrough -> wakeup thread pool-3-thread-57) 00:27:34.351 [DEBUG - New I/O worker #18] (Deferred.java:1330) callback=retry RPC@2136445863 returned Deferred@2070415509(state=PENDING, result=null, callback=(continuation of Deferred@1047173532 after retry RPC@2136445863), errback=(continuation of Deferred@1047173532 after retry RPC@2136445863)), so the following Deferred is getting paused: Deferred@1047173532(state=PAUSED, result=Deferred@2070415509, callback=org.apache.kudu.client.AsyncKuduClient$6@4c174aff -> wakeup thread pool-3-thread-8, errback=passthrough -> wakeup thread pool-3-thread-8) 00:27:34.351 [DEBUG - New I/O worker #470] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.350 [DEBUG - New I/O worker #143] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1592), Traces: [0ms] refreshing cache from master, [141ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [206ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [633ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1079ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.455 [DEBUG - New I/O worker #143] (Deferred.java:1330) callback=retry RPC@1166547347 returned Deferred@1102546426(state=PENDING, result=null, callback=(continuation of Deferred@78975912 after retry RPC@1166547347), errback=(continuation of Deferred@78975912 after retry RPC@1166547347)), so the following Deferred is getting paused: Deferred@78975912(state=PAUSED, result=Deferred@1102546426, callback=org.apache.kudu.client.AsyncKuduClient$6@30ba860a -> wakeup thread pool-3-thread-42, errback=passthrough -> wakeup thread pool-3-thread-42) 00:27:34.341 [DEBUG - New I/O worker #100] (Deferred.java:1330) callback=retry RPC@1708950890 returned Deferred@1642269961(state=PENDING, result=null, callback=(continuation of Deferred@618762065 after retry RPC@1708950890), errback=(continuation of Deferred@618762065 after retry RPC@1708950890)), so the following Deferred is getting paused: Deferred@618762065(state=PAUSED, result=Deferred@1642269961, callback=org.apache.kudu.client.AsyncKuduClient$6@14f90d1a -> wakeup thread pool-3-thread-35, errback=passthrough -> wakeup thread pool-3-thread-35) 00:27:34.330 [DEBUG - New I/O worker #30] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2835), Traces: [0ms] refreshing cache from master, [75ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [76ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [76ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1230ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1936ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [2328ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.456 [DEBUG - New I/O worker #30] (Deferred.java:1330) callback=retry RPC@776924340 returned Deferred@712080805(state=PENDING, result=null, callback=(continuation of Deferred@1869866639 after retry RPC@776924340), errback=(continuation of Deferred@1869866639 after retry RPC@776924340)), so the following Deferred is getting paused: Deferred@1869866639(state=PAUSED, result=Deferred@712080805, callback=org.apache.kudu.client.AsyncKuduClient$6@7247a0a -> wakeup thread pool-3-thread-13, errback=passthrough -> wakeup thread pool-3-thread-13) 00:27:34.326 [DEBUG - New I/O worker #218] (Deferred.java:1330) callback=retry RPC@499486667 returned Deferred@435816619(state=PENDING, result=null, callback=(continuation of Deferred@1559791088 after retry RPC@499486667), errback=(continuation of Deferred@1559791088 after retry RPC@499486667)), so the following Deferred is getting paused: Deferred@1559791088(state=PAUSED, result=Deferred@435816619, callback=org.apache.kudu.client.AsyncKuduClient$6@5c64290f -> wakeup thread pool-3-thread-43, errback=passthrough -> wakeup thread pool-3-thread-43) 00:27:34.322 [DEBUG - New I/O worker #10] (Deferred.java:1330) callback=retry RPC@1464610525 returned Deferred@307145296(state=PENDING, result=null, callback=(continuation of Deferred@1839733913 after retry RPC@1464610525), errback=(continuation of Deferred@1839733913 after retry RPC@1464610525)), so the following Deferred is getting paused: Deferred@1839733913(state=PAUSED, result=Deferred@307145296, callback=org.apache.kudu.client.AsyncKuduClient$6@231699bf -> wakeup thread pool-3-thread-1, errback=passthrough -> wakeup thread pool-3-thread-1) 00:27:34.318 [DEBUG - New I/O worker #91] (Deferred.java:1330) callback=retry RPC@684050090 returned Deferred@1841778134(state=PENDING, result=null, callback=(continuation of Deferred@584806089 after retry RPC@684050090), errback=(continuation of Deferred@584806089 after retry RPC@684050090)), so the following Deferred is getting paused: Deferred@584806089(state=PAUSED, result=Deferred@1841778134, callback=org.apache.kudu.client.AsyncKuduClient$6@5ce08050 -> wakeup thread pool-3-thread-26, errback=passthrough -> wakeup thread pool-3-thread-26) 00:27:34.314 [DEBUG - New I/O worker #141] (Deferred.java:1330) callback=retry RPC@2138169786 returned Deferred@2068740748(state=PENDING, result=null, callback=(continuation of Deferred@1045228417 after retry RPC@2138169786), errback=(continuation of Deferred@1045228417 after retry RPC@2138169786)), so the following Deferred is getting paused: Deferred@1045228417(state=PAUSED, result=Deferred@2068740748, callback=org.apache.kudu.client.AsyncKuduClient$6@7fb43445 -> wakeup thread pool-3-thread-36, errback=passthrough -> wakeup thread pool-3-thread-36) 00:27:34.310 [DEBUG - New I/O worker #356] (Negotiator.java:764) Authenticated connection [id: 0x0cdcfae0, /127.0.0.1:39670 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.309 [DEBUG - New I/O worker #324] (Deferred.java:1330) callback=retry RPC@1601859828 returned Deferred@1531314115(state=PENDING, result=null, callback=(continuation of Deferred@508000975 after retry RPC@1601859828), errback=(continuation of Deferred@508000975 after retry RPC@1601859828)), so the following Deferred is getting paused: Deferred@508000975(state=PAUSED, result=Deferred@1531314115, callback=org.apache.kudu.client.AsyncKuduClient$6@23626db8 -> wakeup thread pool-3-thread-72, errback=passthrough -> wakeup thread pool-3-thread-72) 00:27:34.282 [DEBUG - New I/O worker #17] (Deferred.java:1330) callback=retry RPC@606784960 returned Deferred@538267037(state=PENDING, result=null, callback=(continuation of Deferred@1696057339 after retry RPC@606784960), errback=(continuation of Deferred@1696057339 after retry RPC@606784960)), so the following Deferred is getting paused: Deferred@1696057339(state=PAUSED, result=Deferred@538267037, callback=org.apache.kudu.client.AsyncKuduClient$6@260e5ffa -> wakeup thread pool-3-thread-10, errback=passthrough -> wakeup thread pool-3-thread-10) 00:27:34.265 [DEBUG - New I/O worker #67] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.460 [DEBUG - New I/O worker #67] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2968), Traces: [0ms] refreshing cache from master, [233ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [233ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [233ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [2736ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2773ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [2839ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.460 [DEBUG - New I/O worker #67] (Deferred.java:1330) callback=retry RPC@636868198 returned Deferred@566904320(state=PENDING, result=null, callback=(continuation of Deferred@1690885213 after retry RPC@636868198), errback=(continuation of Deferred@1690885213 after retry RPC@636868198)), so the following Deferred is getting paused: Deferred@1690885213(state=PAUSED, result=Deferred@566904320, callback=org.apache.kudu.client.AsyncKuduClient$6@5e2b3419 -> wakeup thread pool-3-thread-25, errback=passthrough -> wakeup thread pool-3-thread-25) 00:27:34.264 [DEBUG - New I/O worker #137] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.461 [DEBUG - New I/O worker #137] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2496), Traces: [0ms] refreshing cache from master, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [678ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1447ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2300ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.461 [DEBUG - New I/O worker #137] (Deferred.java:1330) callback=retry RPC@1486242690 returned Deferred@1554631752(state=PENDING, result=null, callback=(continuation of Deferred@430655929 after retry RPC@1486242690), errback=(continuation of Deferred@430655929 after retry RPC@1486242690)), so the following Deferred is getting paused: Deferred@430655929(state=PAUSED, result=Deferred@1554631752, callback=org.apache.kudu.client.AsyncKuduClient$6@429845d3 -> wakeup thread pool-3-thread-37, errback=passthrough -> wakeup thread pool-3-thread-37) 00:27:34.263 [DEBUG - New I/O worker #78] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.462 [DEBUG - New I/O worker #78] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2972), Traces: [0ms] refreshing cache from master, [75ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [76ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [76ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1038ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2333ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2774ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.463 [DEBUG - New I/O worker #78] (Deferred.java:1330) callback=retry RPC@666457771 returned Deferred@596039300(state=PENDING, result=null, callback=(continuation of Deferred@1719950480 after retry RPC@666457771), errback=(continuation of Deferred@1719950480 after retry RPC@666457771)), so the following Deferred is getting paused: Deferred@1719950480(state=PAUSED, result=Deferred@596039300, callback=org.apache.kudu.client.AsyncKuduClient$6@3a49dca9 -> wakeup thread pool-3-thread-20, errback=passthrough -> wakeup thread pool-3-thread-20) 00:27:34.263 [DEBUG - New I/O worker #306] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.262 [DEBUG - New I/O worker #15] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.464 [DEBUG - New I/O worker #15] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.464 [DEBUG - New I/O worker #15] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.464 [DEBUG - New I/O worker #15] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.262 [DEBUG - New I/O worker #292] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.465 [DEBUG - New I/O worker #292] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2387), Traces: [0ms] refreshing cache from master, [220ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [508ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1331ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2184ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.466 [DEBUG - New I/O worker #292] (Deferred.java:1330) callback=retry RPC@577245012 returned Deferred@643276737(state=PENDING, result=null, callback=(continuation of Deferred@1666516335 after retry RPC@577245012), errback=(continuation of Deferred@1666516335 after retry RPC@577245012)), so the following Deferred is getting paused: Deferred@1666516335(state=PAUSED, result=Deferred@643276737, callback=org.apache.kudu.client.AsyncKuduClient$6@300c6fc8 -> wakeup thread pool-3-thread-48, errback=passthrough -> wakeup thread pool-3-thread-48) 00:27:34.242 [DEBUG - New I/O worker #326] (Negotiator.java:764) Authenticated connection [id: 0x80f1648e, /127.0.0.1:53040 => /127.22.19.254:44849] using SASL/PLAIN 00:27:34.236 [DEBUG - New I/O worker #116] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.468 [DEBUG - New I/O worker #116] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2509), Traces: [0ms] refreshing cache from master, [13ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [14ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [14ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1582ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2262ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2277ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.468 [DEBUG - New I/O worker #116] (Deferred.java:1330) callback=retry RPC@1855894489 returned Deferred@1788947154(state=PENDING, result=null, callback=(continuation of Deferred@799259106 after retry RPC@1855894489), errback=(continuation of Deferred@799259106 after retry RPC@1855894489)), so the following Deferred is getting paused: Deferred@799259106(state=PAUSED, result=Deferred@1788947154, callback=org.apache.kudu.client.AsyncKuduClient$6@55da9e5 -> wakeup thread pool-3-thread-29, errback=passthrough -> wakeup thread pool-3-thread-29) 00:27:34.234 [DEBUG - New I/O worker #266] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.469 [DEBUG - New I/O worker #266] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1731), Traces: [0ms] refreshing cache from master, [907ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [907ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [907ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [944ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1269ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1497ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.470 [DEBUG - New I/O worker #266] (Deferred.java:1330) callback=retry RPC@1794888125 returned Deferred@1858361402(state=PENDING, result=null, callback=(continuation of Deferred@734452614 after retry RPC@1794888125), errback=(continuation of Deferred@734452614 after retry RPC@1794888125)), so the following Deferred is getting paused: Deferred@734452614(state=PAUSED, result=Deferred@1858361402, callback=org.apache.kudu.client.AsyncKuduClient$6@2e651bd7 -> wakeup thread pool-3-thread-60, errback=passthrough -> wakeup thread pool-3-thread-60) 00:27:34.233 [DEBUG - New I/O worker #217] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.231 [DEBUG - New I/O worker #247] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.231 [DEBUG - New I/O worker #110] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.471 [DEBUG - New I/O worker #110] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1727), Traces: [0ms] refreshing cache from master, [346ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [901ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [901ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [970ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [971ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1483ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.472 [DEBUG - New I/O worker #13] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.230 [DEBUG - New I/O worker #275] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.472 [DEBUG - New I/O worker #247] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1989), Traces: [0ms] refreshing cache from master, [728ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [728ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [728ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1081ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1575ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1750ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.473 [DEBUG - New I/O worker #247] (Deferred.java:1330) callback=retry RPC@873359716 returned Deferred@808577853(state=PENDING, result=null, callback=(continuation of Deferred@1966302047 after retry RPC@873359716), errback=(continuation of Deferred@1966302047 after retry RPC@873359716)), so the following Deferred is getting paused: Deferred@1966302047(state=PAUSED, result=Deferred@808577853, callback=org.apache.kudu.client.AsyncKuduClient$6@30e4e5c -> wakeup thread pool-3-thread-68, errback=passthrough -> wakeup thread pool-3-thread-68) 00:27:34.230 [DEBUG - New I/O worker #71] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.221 [DEBUG - New I/O worker #246] (Deferred.java:1330) callback=retry RPC@1229047364 returned Deferred@1300122548(state=PENDING, result=null, callback=(continuation of Deferred@142396543 after retry RPC@1229047364), errback=(continuation of Deferred@142396543 after retry RPC@1229047364)), so the following Deferred is getting paused: Deferred@142396543(state=PAUSED, result=Deferred@1300122548, callback=org.apache.kudu.client.AsyncKuduClient$6@7e0a4db4 -> wakeup thread pool-3-thread-69, errback=passthrough -> wakeup thread pool-3-thread-69) 00:27:34.499 [DEBUG - New I/O worker #19] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.499 [DEBUG - New I/O worker #17] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.215 [DEBUG - New I/O worker #35] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.500 [DEBUG - New I/O worker #35] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2534), Traces: [0ms] refreshing cache from master, [327ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [417ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [418ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1093ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1573ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2250ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.501 [DEBUG - New I/O worker #35] (Deferred.java:1330) callback=retry RPC@1342083359 returned Deferred@1270946872(state=PENDING, result=null, callback=(continuation of Deferred@247700260 after retry RPC@1342083359), errback=(continuation of Deferred@247700260 after retry RPC@1342083359)), so the following Deferred is getting paused: Deferred@247700260(state=PAUSED, result=Deferred@1270946872, callback=org.apache.kudu.client.AsyncKuduClient$6@44a2e83 -> wakeup thread pool-3-thread-12, errback=passthrough -> wakeup thread pool-3-thread-12) 00:27:34.501 [DEBUG - New I/O worker #100] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.202 [DEBUG - New I/O worker #79] (Deferred.java:1330) callback=retry RPC@459292129 returned Deferred@526367317(state=PENDING, result=null, callback=(continuation of Deferred@1516058586 after retry RPC@459292129), errback=(continuation of Deferred@1516058586 after retry RPC@459292129)), so the following Deferred is getting paused: Deferred@1516058586(state=PAUSED, result=Deferred@526367317, callback=org.apache.kudu.client.AsyncKuduClient$6@71d8eb4d -> wakeup thread pool-3-thread-21, errback=passthrough -> wakeup thread pool-3-thread-21) 00:27:34.502 [DEBUG - New I/O worker #79] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.503 [DEBUG - New I/O worker #141] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.503 [DEBUG - New I/O worker #67] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.192 [DEBUG - New I/O worker #92] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.179 [DEBUG - New I/O worker #305] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.504 [DEBUG - New I/O worker #92] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2538), Traces: [0ms] refreshing cache from master, [330ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [331ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [415ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [755ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1468ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2226ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.177 [DEBUG - New I/O worker #493] (Negotiator.java:764) Authenticated connection [id: 0xe2c3e876, /127.0.0.1:37682 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.172 [DEBUG - New I/O worker #27] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.172 [DEBUG - New I/O worker #660] (Negotiator.java:764) Authenticated connection [id: 0x11ea16b7, /127.0.0.1:39658 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.172 [DEBUG - New I/O worker #485] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.551 [DEBUG - New I/O worker #218] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.551 [DEBUG - New I/O worker #273] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.168 [DEBUG - New I/O worker #478] (Deferred.java:1330) callback=retry RPC@253034657 returned Deferred@187340187(state=PENDING, result=null, callback=(continuation of Deferred@1311247002 after retry RPC@253034657), errback=(continuation of Deferred@1311247002 after retry RPC@253034657)), so the following Deferred is getting paused: Deferred@1311247002(state=PAUSED, result=Deferred@187340187, callback=org.apache.kudu.client.AsyncKuduClient$6@589b9012 -> wakeup thread pool-3-thread-75, errback=passthrough -> wakeup thread pool-3-thread-75) 00:27:34.552 [DEBUG - New I/O worker #134] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.552 [DEBUG - New I/O worker #137] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.164 [DEBUG - New I/O worker #277] (Negotiator.java:764) Authenticated connection [id: 0x786826e5, /127.0.0.1:37706 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.552 [DEBUG - New I/O worker #268] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.553 [DEBUG - New I/O worker #362] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.553 [DEBUG - New I/O worker #78] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.554 [DEBUG - New I/O worker #81] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.554 [DEBUG - New I/O worker #81] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.554 [DEBUG - New I/O worker #248] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.555 [DEBUG - New I/O worker #10] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.160 [DEBUG - New I/O worker #291] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1380), Traces: [0ms] refreshing cache from master, [962ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [962ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [1022ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1228ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1253ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1380ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.555 [DEBUG - New I/O worker #291] (Deferred.java:1330) callback=retry RPC@53597870 returned Deferred@118379024(state=PENDING, result=null, callback=(continuation of Deferred@1108139157 after retry RPC@53597870), errback=(continuation of Deferred@1108139157 after retry RPC@53597870)), so the following Deferred is getting paused: Deferred@1108139157(state=PAUSED, result=Deferred@118379024, callback=org.apache.kudu.client.AsyncKuduClient$6@7e0dd08d -> wakeup thread pool-3-thread-46, errback=passthrough -> wakeup thread pool-3-thread-46) 00:27:34.158 [DEBUG - New I/O worker #70] (Deferred.java:1330) callback=retry RPC@1807516822 returned Deferred@1870863673(state=PENDING, result=null, callback=(continuation of Deferred@713133741 after retry RPC@1807516822), errback=(continuation of Deferred@713133741 after retry RPC@1807516822)), so the following Deferred is getting paused: Deferred@713133741(state=PAUSED, result=Deferred@1870863673, callback=org.apache.kudu.client.AsyncKuduClient$6@14f66c8a -> wakeup thread pool-3-thread-23, errback=passthrough -> wakeup thread pool-3-thread-23) 00:27:34.556 [DEBUG - New I/O worker #322] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.557 [DEBUG - New I/O worker #70] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.145 [DEBUG - New I/O worker #359] (Negotiator.java:764) Authenticated connection [id: 0xeb4dafe1, /127.0.0.1:53052 => /127.22.19.254:44849] using SASL/PLAIN 00:27:34.557 [DEBUG - New I/O worker #92] (Deferred.java:1330) callback=retry RPC@971192941 returned Deferred@1037871497(state=PENDING, result=null, callback=(continuation of Deferred@2027827286 after retry RPC@971192941), errback=(continuation of Deferred@2027827286 after retry RPC@971192941)), so the following Deferred is getting paused: Deferred@2027827286(state=PAUSED, result=Deferred@1037871497, callback=org.apache.kudu.client.AsyncKuduClient$6@69fc16be -> wakeup thread pool-3-thread-28, errback=passthrough -> wakeup thread pool-3-thread-28) 00:27:34.132 [DEBUG - New I/O worker #272] (Deferred.java:1330) callback=retry RPC@458310423 returned Deferred@527352997(state=PENDING, result=null, callback=(continuation of Deferred@1517046060 after retry RPC@458310423), errback=(continuation of Deferred@1517046060 after retry RPC@458310423)), so the following Deferred is getting paused: Deferred@1517046060(state=PAUSED, result=Deferred@527352997, callback=org.apache.kudu.client.AsyncKuduClient$6@7aa4adb3 -> wakeup thread pool-3-thread-55, errback=passthrough -> wakeup thread pool-3-thread-55) 00:27:34.558 [DEBUG - New I/O worker #116] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.558 [DEBUG - New I/O worker #272] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.130 [DEBUG - New I/O worker #271] (Deferred.java:1330) callback=retry RPC@1808358968 returned Deferred@1878443929(state=PENDING, result=null, callback=(continuation of Deferred@720655363 after retry RPC@1808358968), errback=(continuation of Deferred@720655363 after retry RPC@1808358968)), so the following Deferred is getting paused: Deferred@720655363(state=PAUSED, result=Deferred@1878443929, callback=org.apache.kudu.client.AsyncKuduClient$6@3a8dcc50 -> wakeup thread pool-3-thread-50, errback=passthrough -> wakeup thread pool-3-thread-50) 00:27:34.559 [DEBUG - New I/O worker #48] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.560 [DEBUG - New I/O worker #35] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.560 [DEBUG - New I/O worker #306] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.561 [DEBUG - New I/O worker #306] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.561 [DEBUG - New I/O worker #306] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.561 [DEBUG - New I/O worker #142] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.125 [DEBUG - New I/O worker #274] (Deferred.java:1330) callback=retry RPC@1627245321 returned Deferred@1690457038(state=PENDING, result=null, callback=(continuation of Deferred@566280498 after retry RPC@1627245321), errback=(continuation of Deferred@566280498 after retry RPC@1627245321)), so the following Deferred is getting paused: Deferred@566280498(state=PAUSED, result=Deferred@1690457038, callback=org.apache.kudu.client.AsyncKuduClient$6@1e80e27c -> wakeup thread pool-3-thread-52, errback=passthrough -> wakeup thread pool-3-thread-52) 00:27:34.562 [DEBUG - New I/O worker #49] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.562 [DEBUG - New I/O worker #274] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.563 [DEBUG - New I/O worker #493] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.120 [DEBUG - New I/O worker #360] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.572 [DEBUG - New I/O worker #248] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=3067), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1029ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2065ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [3067ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.109 [DEBUG - New I/O worker #264] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.572 [DEBUG - New I/O worker #248] (Deferred.java:1330) callback=retry RPC@579901076 returned Deferred@649008670(state=PENDING, result=null, callback=(continuation of Deferred@1672322223 after retry RPC@579901076), errback=(continuation of Deferred@1672322223 after retry RPC@579901076)), so the following Deferred is getting paused: Deferred@1672322223(state=PAUSED, result=Deferred@649008670, callback=org.apache.kudu.client.AsyncKuduClient$6@740621b9 -> wakeup thread pool-3-thread-67, errback=passthrough -> wakeup thread pool-3-thread-67) 00:27:34.100 [DEBUG - New I/O worker #482] (Negotiator.java:764) Authenticated connection [id: 0x790bef89, /127.0.0.1:39682 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.098 [DEBUG - New I/O worker #22] (Deferred.java:1330) callback=retry RPC@235693129 returned Deferred@171172176(state=PENDING, result=null, callback=(continuation of Deferred@1328639602 after retry RPC@235693129), errback=(continuation of Deferred@1328639602 after retry RPC@235693129)), so the following Deferred is getting paused: Deferred@1328639602(state=PAUSED, result=Deferred@171172176, callback=org.apache.kudu.client.AsyncKuduClient$6@1377773e -> wakeup thread pool-3-thread-7, errback=passthrough -> wakeup thread pool-3-thread-7) 00:27:34.574 [DEBUG - New I/O worker #22] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.080 [DEBUG - New I/O worker #586] (Negotiator.java:764) Authenticated connection [id: 0xe88435c5, /127.0.0.1:53028 => /127.22.19.254:44849] using SASL/PLAIN 00:27:34.050 [DEBUG - New I/O worker #260] (Deferred.java:1330) callback=retry RPC@1866834336 returned Deferred@1803167888(state=PENDING, result=null, callback=(continuation of Deferred@779660187 after retry RPC@1866834336), errback=(continuation of Deferred@779660187 after retry RPC@1866834336)), so the following Deferred is getting paused: Deferred@779660187(state=PAUSED, result=Deferred@1803167888, callback=org.apache.kudu.client.AsyncKuduClient$6@5d1670f7 -> wakeup thread pool-3-thread-63, errback=passthrough -> wakeup thread pool-3-thread-63) 00:27:34.576 [DEBUG - New I/O worker #247] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.037 [DEBUG - New I/O worker #252] (Deferred.java:1330) callback=retry RPC@698806740 returned Deferred@765030456(state=PENDING, result=null, callback=(continuation of Deferred@1755053039 after retry RPC@698806740), errback=(continuation of Deferred@1755053039 after retry RPC@698806740)), so the following Deferred is getting paused: Deferred@1755053039(state=PAUSED, result=Deferred@765030456, callback=org.apache.kudu.client.AsyncKuduClient$6@478f16ef -> wakeup thread pool-3-thread-66, errback=passthrough -> wakeup thread pool-3-thread-66) 00:27:34.033 [DEBUG - New I/O worker #490] (Negotiator.java:764) Authenticated connection [id: 0x97db76ae, /127.0.0.1:53022 => /127.22.19.254:44849] using SASL/PLAIN 00:27:34.583 [DEBUG - New I/O worker #30] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.032 [DEBUG - New I/O worker #472] (Negotiator.java:764) Authenticated connection [id: 0xf1117035, /127.0.0.1:37676 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.588 [DEBUG - New I/O worker #248] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.586 [DEBUG - New I/O worker #144] (Negotiator.java:764) Authenticated connection [id: 0x30e44031, /127.0.0.1:53034 => /127.22.19.254:44849] using SASL/PLAIN 00:27:34.585 [DEBUG - New I/O worker #252] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.589 [DEBUG - New I/O worker #299] (Negotiator.java:764) Authenticated connection [id: 0xa2dda790, /127.0.0.1:53046 => /127.22.19.254:44849] using SASL/PLAIN 00:27:34.584 [DEBUG - New I/O worker #270] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.584 [DEBUG - New I/O worker #46] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.583 [DEBUG - New I/O worker #76] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.591 [DEBUG - New I/O worker #92] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.583 [DEBUG - New I/O worker #220] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.582 [DEBUG - New I/O worker #266] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.577 [DEBUG - New I/O worker #260] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.572 [DEBUG - New I/O worker #264] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1832), Traces: [0ms] refreshing cache from master, [899ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [900ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [900ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1204ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1369ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1498ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:34.632 [DEBUG - New I/O worker #112] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.632 [DEBUG - New I/O worker #112] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.632 [DEBUG - New I/O worker #112] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.633 [DEBUG - New I/O worker #112] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.633 [DEBUG - New I/O worker #247] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.633 [DEBUG - New I/O worker #247] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.634 [DEBUG - New I/O worker #247] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.634 [DEBUG - New I/O worker #470] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.634 [DEBUG - New I/O worker #470] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.634 [DEBUG - New I/O worker #470] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.635 [DEBUG - New I/O worker #470] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.635 [DEBUG - New I/O worker #470] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.637 [DEBUG - New I/O worker #272] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.638 [DEBUG - New I/O worker #272] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.638 [DEBUG - New I/O worker #272] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.638 [DEBUG - New I/O worker #272] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.572 [DEBUG - New I/O worker #360] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1793), Traces: [0ms] refreshing cache from master, [950ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [950ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [951ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1020ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1053ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1340ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.660 [DEBUG - New I/O worker #360] (Deferred.java:1330) callback=retry RPC@304107136 returned Deferred@371185692(state=PENDING, result=null, callback=(continuation of Deferred@1394427067 after retry RPC@304107136), errback=(continuation of Deferred@1394427067 after retry RPC@304107136)), so the following Deferred is getting paused: Deferred@1394427067(state=PAUSED, result=Deferred@371185692, callback=org.apache.kudu.client.AsyncKuduClient$6@7240f167 -> wakeup thread pool-3-thread-74, errback=passthrough -> wakeup thread pool-3-thread-74) 00:27:34.572 [DEBUG - New I/O worker #493] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=843), Traces: [0ms] refreshing cache from master, [114ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [115ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [115ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [833ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.564 [DEBUG - New I/O worker #108] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.563 [DEBUG - New I/O worker #292] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.662 [DEBUG - New I/O worker #273] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.662 [DEBUG - New I/O worker #273] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.562 [DEBUG - New I/O worker #49] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.663 [DEBUG - New I/O worker #49] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.562 [DEBUG - New I/O worker #142] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.561 [DEBUG - New I/O worker #306] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.664 [DEBUG - New I/O worker #306] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.664 [DEBUG - New I/O worker #49] (NetUtil.java:106) Resolved IP of `127.22.19.193' to /127.22.19.193 in 929485ns 00:27:34.664 [DEBUG - New I/O worker #49] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.664 [DEBUG - New I/O worker #100] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.665 [DEBUG - New I/O worker #100] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.665 [DEBUG - New I/O worker #220] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.665 [DEBUG - New I/O worker #220] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.666 [DEBUG - New I/O worker #362] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.666 [DEBUG - New I/O worker #362] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.666 [DEBUG - New I/O worker #362] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.667 [DEBUG - New I/O worker #362] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.667 [DEBUG - New I/O worker #274] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.667 [DEBUG - New I/O worker #274] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.668 [DEBUG - New I/O worker #274] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.668 [DEBUG - New I/O worker #218] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.668 [DEBUG - New I/O worker #362] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299998, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299998, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299998, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299998, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.669 [DEBUG - New I/O worker #46] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.669 [DEBUG - New I/O worker #46] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.669 [DEBUG - New I/O worker #46] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.670 [DEBUG - New I/O worker #46] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.670 [DEBUG - New I/O worker #46] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.561 [DEBUG - New I/O worker #143] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.560 [DEBUG - New I/O worker #138] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.671 [DEBUG - New I/O worker #138] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.672 [DEBUG - New I/O worker #138] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.672 [DEBUG - New I/O worker #141] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.672 [DEBUG - New I/O worker #141] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.560 [DEBUG - New I/O worker #18] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.560 [DEBUG - New I/O worker #48] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.673 [DEBUG - New I/O worker #48] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.673 [DEBUG - New I/O worker #48] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.674 [DEBUG - New I/O worker #48] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.559 [DEBUG - New I/O worker #271] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.558 [DEBUG - New I/O worker #262] (Negotiator.java:764) Authenticated connection [id: 0x2d8de256, /127.0.0.1:37700 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.558 [DEBUG - New I/O worker #139] (Negotiator.java:764) Authenticated connection [id: 0x01569cab, /127.0.0.1:37688 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.557 [DEBUG - New I/O worker #322] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.556 [DEBUG - New I/O worker #267] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.554 [DEBUG - New I/O worker #81] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.553 [DEBUG - New I/O worker #28] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.553 [DEBUG - New I/O worker #268] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.552 [DEBUG - New I/O worker #478] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.551 [DEBUG - New I/O worker #485] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1772), Traces: [0ms] refreshing cache from master, [966ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [966ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [966ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1283ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1393ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1443ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:34.678 [DEBUG - New I/O worker #485] (Deferred.java:1330) callback=retry RPC@1169661191 returned Deferred@1099432297(state=PENDING, result=null, callback=(continuation of Deferred@76191548 after retry RPC@1169661191), errback=(continuation of Deferred@76191548 after retry RPC@1169661191)), so the following Deferred is getting paused: Deferred@76191548(state=PAUSED, result=Deferred@1099432297, callback=org.apache.kudu.client.AsyncKuduClient$6@4c620516 -> wakeup thread pool-3-thread-80, errback=passthrough -> wakeup thread pool-3-thread-80) 00:27:34.550 [DEBUG - New I/O worker #91] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.550 [DEBUG - New I/O worker #246] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.550 [DEBUG - New I/O worker #27] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2561), Traces: [0ms] refreshing cache from master, [45ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [771ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2228ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [2290ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:34.504 [DEBUG - New I/O worker #305] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1725), Traces: [0ms] refreshing cache from master, [964ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [964ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [964ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1390ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1395ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1401ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.680 [DEBUG - New I/O worker #305] (Deferred.java:1330) callback=retry RPC@957860940 returned Deferred@1026056118(state=PENDING, result=null, callback=(continuation of Deferred@2016069239 after retry RPC@957860940), errback=(continuation of Deferred@2016069239 after retry RPC@957860940)), so the following Deferred is getting paused: Deferred@2016069239(state=PAUSED, result=Deferred@1026056118, callback=org.apache.kudu.client.AsyncKuduClient$6@4eeaabb0 -> wakeup thread pool-3-thread-71, errback=passthrough -> wakeup thread pool-3-thread-71) 00:27:34.498 [DEBUG - New I/O worker #324] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.498 [DEBUG - New I/O worker #71] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2980), Traces: [0ms] refreshing cache from master, [49ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [49ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [50ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [748ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1906ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2737ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.681 [DEBUG - New I/O worker #71] (Deferred.java:1330) callback=retry RPC@1652249683 returned Deferred@1715788996(state=PENDING, result=null, callback=(continuation of Deferred@591814248 after retry RPC@1652249683), errback=(continuation of Deferred@591814248 after retry RPC@1652249683)), so the following Deferred is getting paused: Deferred@591814248(state=PAUSED, result=Deferred@1715788996, callback=org.apache.kudu.client.AsyncKuduClient$6@2a4e4ca3 -> wakeup thread pool-3-thread-24, errback=passthrough -> wakeup thread pool-3-thread-24) 00:27:34.465 [DEBUG - New I/O worker #15] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.450 [DEBUG - New I/O worker #47] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.473 [DEBUG - New I/O worker #257] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.472 [DEBUG - New I/O worker #475] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.472 [DEBUG - New I/O worker #275] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1715), Traces: [0ms] refreshing cache from master, [150ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [150ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [151ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [277ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [807ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1473ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.683 [DEBUG - New I/O worker #275] (Deferred.java:1330) callback=retry RPC@2071687551 returned Deferred@2135227080(state=PENDING, result=null, callback=(continuation of Deferred@977697604 after retry RPC@2071687551), errback=(continuation of Deferred@977697604 after retry RPC@2071687551)), so the following Deferred is getting paused: Deferred@977697604(state=PAUSED, result=Deferred@2135227080, callback=org.apache.kudu.client.AsyncKuduClient$6@2b661ce5 -> wakeup thread pool-3-thread-56, errback=passthrough -> wakeup thread pool-3-thread-56) 00:27:34.472 [DEBUG - New I/O worker #110] (Deferred.java:1330) callback=retry RPC@528245276 returned Deferred@457434411(state=PENDING, result=null, callback=(continuation of Deferred@1581344807 after retry RPC@528245276), errback=(continuation of Deferred@1581344807 after retry RPC@528245276)), so the following Deferred is getting paused: Deferred@1581344807(state=PAUSED, result=Deferred@457434411, callback=org.apache.kudu.client.AsyncKuduClient$6@13c5fce2 -> wakeup thread pool-3-thread-31, errback=passthrough -> wakeup thread pool-3-thread-31) 00:27:34.472 [DEBUG - New I/O worker #72] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.471 [DEBUG - New I/O worker #11] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.684 [DEBUG - New I/O worker #110] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.471 [DEBUG - New I/O worker #253] (Negotiator.java:764) Authenticated connection [id: 0x40bfffed, /127.0.0.1:37694 => /127.22.19.252:32833] using SASL/PLAIN 00:27:34.471 [DEBUG - New I/O worker #217] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2386), Traces: [0ms] refreshing cache from master, [179ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [180ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [314ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [560ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1468ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2149ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.696 [DEBUG - New I/O worker #485] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.697 [DEBUG - New I/O worker #217] (Deferred.java:1330) callback=retry RPC@824235548 returned Deferred@891245792(state=PENDING, result=null, callback=(continuation of Deferred@1881005095 after retry RPC@824235548), errback=(continuation of Deferred@1881005095 after retry RPC@824235548)), so the following Deferred is getting paused: Deferred@1881005095(state=PAUSED, result=Deferred@891245792, callback=org.apache.kudu.client.AsyncKuduClient$6@7680760b -> wakeup thread pool-3-thread-44, errback=passthrough -> wakeup thread pool-3-thread-44) 00:27:34.697 [DEBUG - New I/O worker #35] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.698 [DEBUG - New I/O worker #35] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.695 [DEBUG - New I/O worker #248] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.708 [DEBUG - New I/O worker #248] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.708 [DEBUG - New I/O worker #248] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.708 [DEBUG - New I/O worker #248] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.709 [DEBUG - New I/O worker #248] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.694 [DEBUG - New I/O worker #260] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.694 [DEBUG - New I/O worker #108] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.680 [DEBUG - New I/O worker #27] (Deferred.java:1330) callback=retry RPC@2031572248 returned Deferred@2099826402(state=PENDING, result=null, callback=(continuation of Deferred@942300963 after retry RPC@2031572248), errback=(continuation of Deferred@942300963 after retry RPC@2031572248)), so the following Deferred is getting paused: Deferred@942300963(state=PAUSED, result=Deferred@2099826402, callback=org.apache.kudu.client.AsyncKuduClient$6@4a248181 -> wakeup thread pool-3-thread-11, errback=passthrough -> wakeup thread pool-3-thread-11) 00:27:34.679 [DEBUG - New I/O worker #322] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.710 [DEBUG - New I/O worker #322] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.710 [DEBUG - New I/O worker #322] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.679 [DEBUG - New I/O worker #267] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.712 [DEBUG - New I/O worker #267] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.712 [DEBUG - New I/O worker #267] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.712 [DEBUG - New I/O worker #267] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.679 [DEBUG - New I/O worker #81] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.713 [DEBUG - New I/O worker #81] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.714 [DEBUG - New I/O worker #305] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.679 [DEBUG - New I/O worker #292] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.715 [DEBUG - New I/O worker #292] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.718 [DEBUG - New I/O worker #292] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.719 [DEBUG - New I/O worker #292] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.719 [DEBUG - New I/O worker #257] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.679 [DEBUG - New I/O worker #268] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.679 [DEBUG - New I/O worker #92] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.672 [DEBUG - New I/O worker #141] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.720 [DEBUG - New I/O worker #141] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.721 [DEBUG - New I/O worker #141] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.672 [DEBUG - New I/O worker #76] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.672 [DEBUG - New I/O worker #138] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.671 [DEBUG - New I/O worker #10] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.757 [DEBUG - New I/O worker #10] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.757 [DEBUG - New I/O worker #10] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.758 [DEBUG - New I/O worker #10] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.758 [DEBUG - New I/O worker #10] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.670 [DEBUG - New I/O worker #270] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.759 [DEBUG - New I/O worker #270] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.760 [DEBUG - New I/O worker #270] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.761 [DEBUG - New I/O worker #270] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.761 [DEBUG - New I/O worker #270] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.670 [DEBUG - New I/O worker #30] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.762 [DEBUG - New I/O worker #30] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.763 [DEBUG - New I/O worker #30] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.669 [DEBUG - New I/O worker #137] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.763 [DEBUG - New I/O worker #137] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.669 [DEBUG - New I/O worker #134] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.668 [DEBUG - New I/O worker #218] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.764 [DEBUG - New I/O worker #218] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.718 [DEBUG - New I/O worker #47] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.718 [DEBUG - New I/O worker #46] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.765 [DEBUG - New I/O worker #474] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.767 [DEBUG - New I/O worker #387] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.719 [DEBUG - New I/O worker #362] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.668 [DEBUG - New I/O worker #274] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.768 [DEBUG - New I/O worker #274] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.667 [DEBUG - New I/O worker #79] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.768 [DEBUG - New I/O worker #79] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.768 [DEBUG - pool-3-thread-70] (Deferred.java:1330) callback=disconnect callback@1019056843 returned Deferred@2053361779(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@696401819 after disconnect callback@1019056843), errback=passthrough -> (continuation of Deferred@696401819 after disconnect callback@1019056843)), so the following Deferred is getting paused: Deferred@696401819(state=PAUSED, result=Deferred@2053361779, callback=, errback=) 00:27:34.768 [DEBUG - New I/O worker #79] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.769 [DEBUG - New I/O worker #79] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.769 [DEBUG - New I/O worker #79] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.666 [DEBUG - New I/O worker #220] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.770 [DEBUG - New I/O worker #220] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.665 [DEBUG - New I/O worker #100] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.770 [DEBUG - New I/O worker #220] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.664 [DEBUG - New I/O worker #49] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.664 [DEBUG - New I/O worker #17] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.771 [DEBUG - New I/O worker #17] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.772 [DEBUG - New I/O worker #17] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.772 [DEBUG - pool-3-thread-47] (Deferred.java:1330) callback=disconnect callback@282897966 returned Deferred@27440247(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@750139445 after disconnect callback@282897966), errback=passthrough -> (continuation of Deferred@750139445 after disconnect callback@282897966)), so the following Deferred is getting paused: Deferred@750139445(state=PAUSED, result=Deferred@27440247, callback=, errback=) 00:27:34.772 [DEBUG - New I/O worker #17] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.772 [DEBUG - New I/O worker #17] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.773 [DEBUG - pool-3-thread-73] (Deferred.java:1330) callback=disconnect callback@469738925 returned Deferred@967282537(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2135781855 after disconnect callback@469738925), errback=passthrough -> (continuation of Deferred@2135781855 after disconnect callback@469738925)), so the following Deferred is getting paused: Deferred@2135781855(state=PAUSED, result=Deferred@967282537, callback=, errback=) 00:27:34.773 [DEBUG - pool-3-thread-19] (Deferred.java:1330) callback=disconnect callback@1364659206 returned Deferred@1605782156(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@795053075 after disconnect callback@1364659206), errback=passthrough -> (continuation of Deferred@795053075 after disconnect callback@1364659206)), so the following Deferred is getting paused: Deferred@795053075(state=PAUSED, result=Deferred@1605782156, callback=, errback=) 00:27:34.773 [DEBUG - pool-3-thread-67] (Deferred.java:1330) callback=disconnect callback@1511009100 returned Deferred@1281664956(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1906379340 after disconnect callback@1511009100), errback=passthrough -> (continuation of Deferred@1906379340 after disconnect callback@1511009100)), so the following Deferred is getting paused: Deferred@1906379340(state=PAUSED, result=Deferred@1281664956, callback=, errback=) 00:27:34.774 [DEBUG - New I/O worker #137] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.774 [DEBUG - New I/O worker #137] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.774 [DEBUG - pool-3-thread-10] (Deferred.java:1330) callback=disconnect callback@48677417 returned Deferred@1152918516(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@224819005 after disconnect callback@48677417), errback=passthrough -> (continuation of Deferred@224819005 after disconnect callback@48677417)), so the following Deferred is getting paused: Deferred@224819005(state=PAUSED, result=Deferred@1152918516, callback=, errback=) 00:27:34.663 [DEBUG - New I/O worker #142] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.663 [DEBUG - New I/O worker #19] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.775 [DEBUG - New I/O worker #19] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.776 [DEBUG - New I/O worker #19] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.776 [DEBUG - New I/O worker #19] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.776 [DEBUG - New I/O worker #30] (NetUtil.java:106) Resolved IP of `127.22.19.194' to /127.22.19.194 in 13100703ns 00:27:34.776 [DEBUG - New I/O worker #142] (NetUtil.java:106) Resolved IP of `127.22.19.194' to /127.22.19.194 in 1519945ns 00:27:34.662 [DEBUG - New I/O worker #273] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.777 [DEBUG - New I/O worker #273] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.777 [DEBUG - New I/O worker #273] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.661 [DEBUG - New I/O worker #493] (Deferred.java:1330) callback=retry RPC@20109050 returned Deferred@84759044(state=PENDING, result=null, callback=(continuation of Deferred@1074781377 after retry RPC@20109050), errback=(continuation of Deferred@1074781377 after retry RPC@20109050)), so the following Deferred is getting paused: Deferred@1074781377(state=PAUSED, result=Deferred@84759044, callback=org.apache.kudu.client.AsyncKuduClient$6@6617732d -> wakeup thread Time-limited test, errback=passthrough -> wakeup thread Time-limited test) 00:27:34.778 [DEBUG - pool-3-thread-57] (Deferred.java:1330) callback=disconnect callback@325631467 returned Deferred@1819610421(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1423407656 after disconnect callback@325631467), errback=passthrough -> (continuation of Deferred@1423407656 after disconnect callback@325631467)), so the following Deferred is getting paused: Deferred@1423407656(state=PAUSED, result=Deferred@1819610421, callback=, errback=) 00:27:34.778 [DEBUG - New I/O worker #493] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.658 [DEBUG - New I/O worker #272] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299980, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299980, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299980, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.658 [DEBUG - New I/O worker #70] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.657 [DEBUG - New I/O worker #266] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.657 [DEBUG - New I/O worker #22] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.807 [DEBUG - New I/O worker #345] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.634 [DEBUG - New I/O worker #247] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.808 [DEBUG - New I/O worker #247] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.633 [DEBUG - New I/O worker #112] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.632 [DEBUG - New I/O worker #264] (Deferred.java:1330) callback=retry RPC@331782647 returned Deferred@402198359(state=PENDING, result=null, callback=(continuation of Deferred@1392219084 after retry RPC@331782647), errback=(continuation of Deferred@1392219084 after retry RPC@331782647)), so the following Deferred is getting paused: Deferred@1392219084(state=PAUSED, result=Deferred@402198359, callback=org.apache.kudu.client.AsyncKuduClient$6@44cdf5e8 -> wakeup thread pool-3-thread-61, errback=passthrough -> wakeup thread pool-3-thread-61) 00:27:34.628 [DEBUG - New I/O worker #105] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.618 [DEBUG - New I/O worker #14] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.811 [DEBUG - New I/O worker #14] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.811 [DEBUG - New I/O worker #14] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.812 [DEBUG - New I/O worker #14] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.812 [DEBUG - New I/O worker #14] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.617 [DEBUG - New I/O worker #269] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.813 [DEBUG - New I/O worker #269] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.813 [DEBUG - New I/O worker #269] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.813 [DEBUG - New I/O worker #269] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.617 [DEBUG - New I/O worker #472] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.814 [DEBUG - New I/O worker #472] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1082), Traces: [0ms] refreshing cache from master, [98ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [98ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [99ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [327ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [853ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [885ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.815 [DEBUG - New I/O worker #472] (Deferred.java:1330) callback=retry RPC@2098144907 returned Deferred@2033237490(state=PENDING, result=null, callback=(continuation of Deferred@1009921200 after retry RPC@2098144907), errback=(continuation of Deferred@1009921200 after retry RPC@2098144907)), so the following Deferred is getting paused: Deferred@1009921200(state=PAUSED, result=Deferred@2033237490, callback=org.apache.kudu.client.AsyncKuduClient$6@97b8172 -> wakeup thread pool-3-thread-78, errback=passthrough -> wakeup thread pool-3-thread-78) 00:27:34.815 [DEBUG - pool-3-thread-5] (Deferred.java:1330) callback=disconnect callback@936548374 returned Deferred@1144165617(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@958919975 after disconnect callback@936548374), errback=passthrough -> (continuation of Deferred@958919975 after disconnect callback@936548374)), so the following Deferred is getting paused: Deferred@958919975(state=PAUSED, result=Deferred@1144165617, callback=, errback=) 00:27:34.617 [DEBUG - New I/O worker #116] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.816 [DEBUG - New I/O worker #116] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.816 [DEBUG - New I/O worker #116] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.816 [DEBUG - New I/O worker #116] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.817 [DEBUG - New I/O worker #116] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.617 [DEBUG - New I/O worker #221] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.817 [DEBUG - New I/O worker #221] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.818 [DEBUG - New I/O worker #221] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.818 [DEBUG - New I/O worker #221] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.818 [DEBUG - New I/O worker #181] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.818 [DEBUG - New I/O worker #105] (NetUtil.java:106) Resolved IP of `127.22.19.195' to /127.22.19.195 in 7469919ns 00:27:34.819 [DEBUG - New I/O worker #105] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.616 [DEBUG - New I/O worker #12] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.819 [DEBUG - New I/O worker #12] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.819 [DEBUG - New I/O worker #12] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.820 [DEBUG - New I/O worker #12] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.820 [DEBUG - New I/O worker #12] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.616 [DEBUG - New I/O worker #277] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.821 [DEBUG - New I/O worker #277] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=993), Traces: [0ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [735ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [760ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [788ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.821 [DEBUG - New I/O worker #21] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.822 [DEBUG - New I/O worker #277] (Deferred.java:1330) callback=retry RPC@1288932158 returned Deferred@1223427708(state=PENDING, result=null, callback=(continuation of Deferred@233734405 after retry RPC@1288932158), errback=(continuation of Deferred@233734405 after retry RPC@1288932158)), so the following Deferred is getting paused: Deferred@233734405(state=PAUSED, result=Deferred@1223427708, callback=org.apache.kudu.client.AsyncKuduClient$6@6cf3f1b9 -> wakeup thread pool-3-thread-51, errback=passthrough -> wakeup thread pool-3-thread-51) 00:27:34.822 [DEBUG - New I/O worker #105] (NetUtil.java:106) Resolved IP of `127.22.19.195' to /127.22.19.195 in 3106349ns 00:27:34.822 [DEBUG - New I/O worker #105] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.822 [DEBUG - New I/O worker #105] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.616 [DEBUG - New I/O worker #102] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.615 [DEBUG - New I/O worker #78] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.823 [DEBUG - New I/O worker #105] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.824 [DEBUG - pool-3-thread-33] (Deferred.java:1330) callback=disconnect callback@1703540629 returned Deferred@103791358(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@725506656 after disconnect callback@1703540629), errback=passthrough -> (continuation of Deferred@725506656 after disconnect callback@1703540629)), so the following Deferred is getting paused: Deferred@725506656(state=PAUSED, result=Deferred@103791358, callback=, errback=) 00:27:34.825 [DEBUG - New I/O worker #106] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.615 [DEBUG - New I/O worker #291] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.614 [DEBUG - New I/O worker #219] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.826 [DEBUG - New I/O worker #219] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.826 [DEBUG - New I/O worker #219] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.826 [DEBUG - New I/O worker #219] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.827 [DEBUG - New I/O worker #219] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.829 [DEBUG - pool-3-thread-49] (Deferred.java:1330) callback=disconnect callback@1988118704 returned Deferred@997393706(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1539098976 after disconnect callback@1988118704), errback=passthrough -> (continuation of Deferred@1539098976 after disconnect callback@1988118704)), so the following Deferred is getting paused: Deferred@1539098976(state=PAUSED, result=Deferred@997393706, callback=, errback=) 00:27:34.614 [DEBUG - New I/O worker #67] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.829 [DEBUG - New I/O worker #67] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.614 [DEBUG - New I/O worker #13] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.830 [DEBUG - New I/O worker #13] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.830 [DEBUG - New I/O worker #13] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.830 [DEBUG - New I/O worker #13] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.831 [DEBUG - New I/O worker #13] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.831 [DEBUG - New I/O worker #346] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.591 [DEBUG - New I/O worker #288] (Negotiator.java:764) Authenticated connection [id: 0xec4e3925, /127.0.0.1:39664 => /127.22.19.253:37879] using SASL/PLAIN 00:27:34.831 [DEBUG - New I/O worker #357] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.832 [DEBUG - pool-3-thread-6] (Deferred.java:1330) callback=disconnect callback@1382570348 returned Deferred@753411970(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1666437352 after disconnect callback@1382570348), errback=passthrough -> (continuation of Deferred@1666437352 after disconnect callback@1382570348)), so the following Deferred is getting paused: Deferred@1666437352(state=PAUSED, result=Deferred@753411970, callback=, errback=) 00:27:34.832 [DEBUG - New I/O worker #81] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.832 [DEBUG - New I/O worker #248] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.833 [DEBUG - New I/O worker #84] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.833 [DEBUG - New I/O worker #42] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.833 [DEBUG - New I/O worker #267] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.834 [DEBUG - New I/O worker #273] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.831 [DEBUG - New I/O worker #123] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.829 [DEBUG - New I/O worker #67] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.829 [DEBUG - New I/O worker #298] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.825 [DEBUG - New I/O worker #277] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.824 [DEBUG - New I/O worker #78] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.823 [DEBUG - New I/O worker #102] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.873 [DEBUG - New I/O worker #102] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.873 [DEBUG - New I/O worker #102] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.874 [DEBUG - New I/O worker #102] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.822 [DEBUG - New I/O worker #472] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.821 [DEBUG - pool-3-thread-2] (Deferred.java:1330) callback=disconnect callback@702679821 returned Deferred@1568192951(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@65721858 after disconnect callback@702679821), errback=passthrough -> (continuation of Deferred@65721858 after disconnect callback@702679821)), so the following Deferred is getting paused: Deferred@65721858(state=PAUSED, result=Deferred@1568192951, callback=, errback=) 00:27:34.818 [DEBUG - New I/O worker #221] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.818 [DEBUG - pool-3-thread-29] (Deferred.java:1330) callback=disconnect callback@365020510 returned Deferred@1342300816(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1918429067 after disconnect callback@365020510), errback=passthrough -> (continuation of Deferred@1918429067 after disconnect callback@365020510)), so the following Deferred is getting paused: Deferred@1918429067(state=PAUSED, result=Deferred@1342300816, callback=, errback=) 00:27:34.876 [DEBUG - New I/O worker #393] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.876 [DEBUG - New I/O worker #25] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.876 [DEBUG - New I/O worker #182] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.876 [DEBUG - New I/O worker #128] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.877 [DEBUG - New I/O worker #322] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.877 [DEBUG - New I/O worker #251] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.877 [DEBUG - New I/O worker #220] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.877 [DEBUG - New I/O worker #301] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.878 [DEBUG - New I/O worker #13] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.817 [DEBUG - New I/O worker #264] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.815 [DEBUG - New I/O worker #20] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.814 [DEBUG - New I/O worker #269] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.812 [DEBUG - New I/O worker #113] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.879 [DEBUG - New I/O worker #293] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.812 [DEBUG - pool-3-thread-30] (Deferred.java:1330) callback=disconnect callback@435264284 returned Deferred@1940217764(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1009110817 after disconnect callback@435264284), errback=passthrough -> (continuation of Deferred@1009110817 after disconnect callback@435264284)), so the following Deferred is getting paused: Deferred@1009110817(state=PAUSED, result=Deferred@1940217764, callback=, errback=) 00:27:34.811 [DEBUG - New I/O worker #420] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.810 [DEBUG - pool-3-thread-68] (Deferred.java:1330) callback=disconnect callback@1474323449 returned Deferred@595317484(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1544627641 after disconnect callback@1474323449), errback=passthrough -> (continuation of Deferred@1544627641 after disconnect callback@1474323449)), so the following Deferred is getting paused: Deferred@1544627641(state=PAUSED, result=Deferred@595317484, callback=, errback=) 00:27:34.808 [DEBUG - New I/O worker #342] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.807 [DEBUG - New I/O worker #22] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.884 [DEBUG - New I/O worker #22] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.885 [DEBUG - New I/O worker #22] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.885 [DEBUG - New I/O worker #493] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.807 [DEBUG - New I/O worker #16] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.807 [DEBUG - New I/O worker #266] (NetUtil.java:106) Resolved IP of `127.22.19.194' to /127.22.19.194 in 27703269ns 00:27:34.886 [DEBUG - New I/O worker #266] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.886 [DEBUG - New I/O worker #266] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.886 [DEBUG - New I/O worker #266] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.887 [DEBUG - New I/O worker #266] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.888 [DEBUG - pool-3-thread-60] (Deferred.java:1330) callback=disconnect callback@1296061655 returned Deferred@1467693602(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1404042478 after disconnect callback@1296061655), errback=passthrough -> (continuation of Deferred@1404042478 after disconnect callback@1296061655)), so the following Deferred is getting paused: Deferred@1404042478(state=PAUSED, result=Deferred@1467693602, callback=, errback=) 00:27:34.888 [DEBUG - New I/O worker #412] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.889 [DEBUG - New I/O worker #414] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.807 [DEBUG - New I/O worker #272] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.806 [DEBUG - pool-3-thread-55] (Deferred.java:1330) callback=disconnect callback@137912128 returned Deferred@430453482(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2004222473 after disconnect callback@137912128), errback=passthrough -> (continuation of Deferred@2004222473 after disconnect callback@137912128)), so the following Deferred is getting paused: Deferred@2004222473(state=PAUSED, result=Deferred@430453482, callback=, errback=) 00:27:34.806 [DEBUG - New I/O worker #305] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.889 [DEBUG - New I/O worker #305] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.890 [DEBUG - New I/O worker #305] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.890 [DEBUG - New I/O worker #305] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.890 [DEBUG - New I/O worker #305] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.779 [DEBUG - New I/O worker #70] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.777 [DEBUG - New I/O worker #142] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.936 [DEBUG - New I/O worker #142] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.937 [DEBUG - pool-3-thread-71] (Deferred.java:1330) callback=disconnect callback@100079406 returned Deferred@1785634177(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@115061604 after disconnect callback@100079406), errback=passthrough -> (continuation of Deferred@115061604 after disconnect callback@100079406)), so the following Deferred is getting paused: Deferred@115061604(state=PAUSED, result=Deferred@1785634177, callback=, errback=) 00:27:34.776 [DEBUG - New I/O worker #30] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.776 [DEBUG - New I/O worker #19] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.775 [DEBUG - pool-3-thread-54] (Deferred.java:1330) callback=disconnect callback@1965929893 returned Deferred@676432048(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1904655193 after disconnect callback@1965929893), errback=passthrough -> (continuation of Deferred@1904655193 after disconnect callback@1965929893)), so the following Deferred is getting paused: Deferred@1904655193(state=PAUSED, result=Deferred@676432048, callback=, errback=) 00:27:34.938 [DEBUG - pool-3-thread-41] (Deferred.java:1330) callback=disconnect callback@1420954210 returned Deferred@1048009333(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1391766542 after disconnect callback@1420954210), errback=passthrough -> (continuation of Deferred@1391766542 after disconnect callback@1420954210)), so the following Deferred is getting paused: Deferred@1391766542(state=PAUSED, result=Deferred@1048009333, callback=, errback=) 00:27:34.938 [DEBUG - New I/O worker #407] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.938 [DEBUG - New I/O worker #290] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.939 [DEBUG - New I/O worker #39] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.939 [DEBUG - New I/O worker #60] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.774 [DEBUG - New I/O worker #137] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.940 [DEBUG - New I/O worker #70] (NetUtil.java:106) Resolved IP of `127.22.19.193' to /127.22.19.193 in 48771975ns 00:27:34.940 [DEBUG - New I/O worker #70] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.940 [DEBUG - New I/O worker #70] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.941 [DEBUG - New I/O worker #146] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.941 [DEBUG - New I/O worker #70] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.941 [DEBUG - New I/O worker #285] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.773 [DEBUG - pool-3-thread-18] (Deferred.java:1330) callback=disconnect callback@869212176 returned Deferred@32446337(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@76979700 after disconnect callback@869212176), errback=passthrough -> (continuation of Deferred@76979700 after disconnect callback@869212176)), so the following Deferred is getting paused: Deferred@76979700(state=PAUSED, result=Deferred@32446337, callback=, errback=) 00:27:34.772 [DEBUG - New I/O worker #134] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.942 [DEBUG - pool-3-thread-23] (Deferred.java:1330) callback=disconnect callback@1600594420 returned Deferred@242170914(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2147206974 after disconnect callback@1600594420), errback=passthrough -> (continuation of Deferred@2147206974 after disconnect callback@1600594420)), so the following Deferred is getting paused: Deferred@2147206974(state=PAUSED, result=Deferred@242170914, callback=, errback=) 00:27:34.772 [DEBUG - pool-3-thread-16] (Deferred.java:1330) callback=disconnect callback@386650888 returned Deferred@885749121(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1129598177 after disconnect callback@386650888), errback=passthrough -> (continuation of Deferred@1129598177 after disconnect callback@386650888)), so the following Deferred is getting paused: Deferred@1129598177(state=PAUSED, result=Deferred@885749121, callback=, errback=) 00:27:34.771 [DEBUG - pool-3-thread-21] (Deferred.java:1330) callback=disconnect callback@1615341318 returned Deferred@623420655(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@441610479 after disconnect callback@1615341318), errback=passthrough -> (continuation of Deferred@441610479 after disconnect callback@1615341318)), so the following Deferred is getting paused: Deferred@441610479(state=PAUSED, result=Deferred@623420655, callback=, errback=) 00:27:34.771 [DEBUG - pool-3-thread-76] (Deferred.java:1330) callback=disconnect callback@953772099 returned Deferred@462831597(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1042074946 after disconnect callback@953772099), errback=passthrough -> (continuation of Deferred@1042074946 after disconnect callback@953772099)), so the following Deferred is getting paused: Deferred@1042074946(state=PAUSED, result=Deferred@462831597, callback=, errback=) 00:27:34.771 [DEBUG - pool-3-thread-15] (Deferred.java:1330) callback=disconnect callback@1413220259 returned Deferred@201410104(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1370492820 after disconnect callback@1413220259), errback=passthrough -> (continuation of Deferred@1370492820 after disconnect callback@1413220259)), so the following Deferred is getting paused: Deferred@1370492820(state=PAUSED, result=Deferred@201410104, callback=, errback=) 00:27:34.770 [DEBUG - New I/O worker #100] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.943 [DEBUG - New I/O worker #100] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.945 [DEBUG - pool-3-thread-35] (Deferred.java:1330) callback=disconnect callback@948305675 returned Deferred@95914599(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@156889215 after disconnect callback@948305675), errback=passthrough -> (continuation of Deferred@156889215 after disconnect callback@948305675)), so the following Deferred is getting paused: Deferred@156889215(state=PAUSED, result=Deferred@95914599, callback=, errback=) 00:27:34.945 [DEBUG - New I/O worker #472] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.945 [DEBUG - New I/O worker #192] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.946 [DEBUG - New I/O worker #231] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.770 [DEBUG - pool-3-thread-52] (Deferred.java:1330) callback=disconnect callback@500229601 returned Deferred@1574266601(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@153629090 after disconnect callback@500229601), errback=passthrough -> (continuation of Deferred@153629090 after disconnect callback@500229601)), so the following Deferred is getting paused: Deferred@153629090(state=PAUSED, result=Deferred@1574266601, callback=, errback=) 00:27:34.769 [DEBUG - pool-3-thread-79] (Deferred.java:1330) callback=disconnect callback@528838578 returned Deferred@1261710910(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1313454446 after disconnect callback@528838578), errback=passthrough -> (continuation of Deferred@1313454446 after disconnect callback@528838578)), so the following Deferred is getting paused: Deferred@1313454446(state=PAUSED, result=Deferred@1261710910, callback=, errback=) 00:27:34.768 [DEBUG - pool-3-thread-17] (Deferred.java:1330) callback=disconnect callback@582102451 returned Deferred@1102776281(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2100332776 after disconnect callback@582102451), errback=passthrough -> (continuation of Deferred@2100332776 after disconnect callback@582102451)), so the following Deferred is getting paused: Deferred@2100332776(state=PAUSED, result=Deferred@1102776281, callback=, errback=) 00:27:34.768 [DEBUG - pool-3-thread-1] (Deferred.java:1330) callback=disconnect callback@138427480 returned Deferred@1070891036(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@512367609 after disconnect callback@138427480), errback=passthrough -> (continuation of Deferred@512367609 after disconnect callback@138427480)), so the following Deferred is getting paused: Deferred@512367609(state=PAUSED, result=Deferred@1070891036, callback=, errback=) 00:27:34.768 [DEBUG - pool-3-thread-4] (Deferred.java:1330) callback=disconnect callback@1135351315 returned Deferred@338076363(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@181723196 after disconnect callback@1135351315), errback=passthrough -> (continuation of Deferred@181723196 after disconnect callback@1135351315)), so the following Deferred is getting paused: Deferred@181723196(state=PAUSED, result=Deferred@338076363, callback=, errback=) 00:27:34.768 [DEBUG - pool-3-thread-53] (Deferred.java:1330) callback=disconnect callback@646077940 returned Deferred@128700178(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@454153793 after disconnect callback@646077940), errback=passthrough -> (continuation of Deferred@454153793 after disconnect callback@646077940)), so the following Deferred is getting paused: Deferred@454153793(state=PAUSED, result=Deferred@128700178, callback=, errback=) 00:27:34.768 [DEBUG - pool-3-thread-36] (Deferred.java:1330) callback=disconnect callback@1808024769 returned Deferred@68890560(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1220435305 after disconnect callback@1808024769), errback=passthrough -> (continuation of Deferred@1220435305 after disconnect callback@1808024769)), so the following Deferred is getting paused: Deferred@1220435305(state=PAUSED, result=Deferred@68890560, callback=, errback=) 00:27:34.719 [DEBUG - New I/O worker #306] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.765 [DEBUG - New I/O worker #82] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.718 [DEBUG - New I/O worker #470] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.764 [DEBUG - New I/O worker #218] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.949 [DEBUG - New I/O worker #218] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.715 [DEBUG - New I/O worker #48] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.763 [DEBUG - New I/O worker #36] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.759 [DEBUG - New I/O worker #74] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.950 [DEBUG - pool-3-thread-43] (Deferred.java:1330) callback=disconnect callback@1203000782 returned Deferred@840847404(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@50671100 after disconnect callback@1203000782), errback=passthrough -> (continuation of Deferred@50671100 after disconnect callback@1203000782)), so the following Deferred is getting paused: Deferred@50671100(state=PAUSED, result=Deferred@840847404, callback=, errback=) 00:27:34.950 [DEBUG - New I/O worker #297] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.757 [DEBUG - New I/O worker #257] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.951 [DEBUG - New I/O worker #257] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.756 [DEBUG - New I/O worker #268] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.951 [DEBUG - New I/O worker #268] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.756 [DEBUG - New I/O worker #92] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.952 [DEBUG - New I/O worker #92] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.952 [DEBUG - New I/O worker #92] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.953 [DEBUG - New I/O worker #92] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.954 [DEBUG - pool-3-thread-58] (Deferred.java:1330) callback=disconnect callback@1035910839 returned Deferred@1961826717(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@622791417 after disconnect callback@1035910839), errback=passthrough -> (continuation of Deferred@622791417 after disconnect callback@1035910839)), so the following Deferred is getting paused: Deferred@622791417(state=PAUSED, result=Deferred@1961826717, callback=, errback=) 00:27:34.755 [DEBUG - New I/O worker #138] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299966, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299966, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299966, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299966, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.754 [DEBUG - New I/O worker #11] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.754 [DEBUG - New I/O worker #27] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.984 [DEBUG - New I/O worker #11] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.985 [DEBUG - New I/O worker #11] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.753 [DEBUG - New I/O worker #478] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.985 [DEBUG - New I/O worker #478] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.985 [DEBUG - New I/O worker #478] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.986 [DEBUG - New I/O worker #478] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.986 [DEBUG - New I/O worker #478] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.753 [DEBUG - New I/O worker #72] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.987 [DEBUG - New I/O worker #72] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.987 [DEBUG - New I/O worker #177] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.988 [DEBUG - pool-3-thread-38] (Deferred.java:1330) callback=disconnect callback@1082535460 returned Deferred@36337975(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2017309925 after disconnect callback@1082535460), errback=passthrough -> (continuation of Deferred@2017309925 after disconnect callback@1082535460)), so the following Deferred is getting paused: Deferred@2017309925(state=PAUSED, result=Deferred@36337975, callback=, errback=) 00:27:34.988 [DEBUG - pool-3-thread-75] (Deferred.java:1330) callback=disconnect callback@8897199 returned Deferred@1766985015(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@883533177 after disconnect callback@8897199), errback=passthrough -> (continuation of Deferred@883533177 after disconnect callback@8897199)), so the following Deferred is getting paused: Deferred@883533177(state=PAUSED, result=Deferred@1766985015, callback=, errback=) 00:27:34.988 [DEBUG - New I/O worker #145] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.988 [DEBUG - New I/O worker #286] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.753 [DEBUG - New I/O worker #110] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.989 [DEBUG - New I/O worker #110] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.989 [DEBUG - New I/O worker #110] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.990 [DEBUG - New I/O worker #110] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.990 [DEBUG - New I/O worker #110] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.753 [DEBUG - New I/O worker #71] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.752 [DEBUG - New I/O worker #475] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.991 [DEBUG - New I/O worker #475] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.991 [DEBUG - New I/O worker #179] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.991 [DEBUG - New I/O worker #92] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.752 [DEBUG - New I/O worker #139] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.751 [DEBUG - New I/O worker #485] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.992 [DEBUG - New I/O worker #485] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.992 [DEBUG - New I/O worker #485] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.751 [DEBUG - New I/O worker #28] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.993 [DEBUG - New I/O worker #28] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.993 [DEBUG - New I/O worker #28] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.993 [DEBUG - New I/O worker #28] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:34.994 [DEBUG - New I/O worker #28] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.751 [DEBUG - New I/O worker #246] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.995 [DEBUG - New I/O worker #246] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.995 [DEBUG - pool-3-thread-14] (Deferred.java:1330) callback=disconnect callback@1122963086 returned Deferred@853507567(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@743568283 after disconnect callback@1122963086), errback=passthrough -> (continuation of Deferred@743568283 after disconnect callback@1122963086)), so the following Deferred is getting paused: Deferred@743568283(state=PAUSED, result=Deferred@853507567, callback=, errback=) 00:27:34.751 [DEBUG - New I/O worker #271] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.002 [DEBUG - New I/O worker #271] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.002 [DEBUG - New I/O worker #271] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.002 [DEBUG - New I/O worker #271] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.003 [DEBUG - New I/O worker #271] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.751 [DEBUG - New I/O worker #324] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.003 [DEBUG - New I/O worker #324] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.003 [DEBUG - New I/O worker #324] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.004 [DEBUG - New I/O worker #324] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.004 [DEBUG - New I/O worker #324] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.007 [DEBUG - New I/O worker #71] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.751 [DEBUG - New I/O worker #91] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.750 [DEBUG - New I/O worker #217] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:34.750 [DEBUG - New I/O worker #253] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:34.750 [DEBUG - New I/O worker #143] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.009 [DEBUG - New I/O worker #253] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2577), Traces: [0ms] refreshing cache from master, [1477ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [1477ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [1477ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [2041ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2141ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2318ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:34.722 [DEBUG - New I/O worker #76] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.719 [DEBUG - New I/O worker #292] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.716 [DEBUG - New I/O worker #360] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.010 [DEBUG - New I/O worker #91] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.011 [DEBUG - New I/O worker #91] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.011 [DEBUG - New I/O worker #91] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.012 [DEBUG - New I/O worker #91] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.013 [DEBUG - pool-3-thread-26] (Deferred.java:1330) callback=disconnect callback@542956588 returned Deferred@545576200(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1971219446 after disconnect callback@542956588), errback=passthrough -> (continuation of Deferred@1971219446 after disconnect callback@542956588)), so the following Deferred is getting paused: Deferred@1971219446(state=PAUSED, result=Deferred@545576200, callback=, errback=) 00:27:35.013 [DEBUG - New I/O worker #133] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.014 [DEBUG - New I/O worker #117] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.715 [DEBUG - New I/O worker #18] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.715 [DEBUG - New I/O worker #252] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.014 [DEBUG - New I/O worker #252] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.015 [DEBUG - New I/O worker #252] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.714 [DEBUG - New I/O worker #15] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.712 [DEBUG - New I/O worker #275] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.015 [DEBUG - pool-3-thread-50] (Deferred.java:1330) callback=disconnect callback@1958821859 returned Deferred@740081829(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@832788056 after disconnect callback@1958821859), errback=passthrough -> (continuation of Deferred@832788056 after disconnect callback@1958821859)), so the following Deferred is getting paused: Deferred@832788056(state=PAUSED, result=Deferred@740081829, callback=, errback=) 00:27:34.711 [DEBUG - New I/O worker #35] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.016 [DEBUG - New I/O worker #466] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.016 [DEBUG - New I/O worker #252] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.017 [DEBUG - New I/O worker #252] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.017 [DEBUG - New I/O worker #324] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.018 [DEBUG - pool-3-thread-66] (Deferred.java:1330) callback=disconnect callback@1481885769 returned Deferred@1079431876(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1049355928 after disconnect callback@1481885769), errback=passthrough -> (continuation of Deferred@1049355928 after disconnect callback@1481885769)), so the following Deferred is getting paused: Deferred@1049355928(state=PAUSED, result=Deferred@1079431876, callback=, errback=) 00:27:35.018 [DEBUG - New I/O worker #302] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.019 [DEBUG - New I/O worker #448] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.019 [DEBUG - New I/O worker #271] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.710 [DEBUG - New I/O worker #108] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.709 [DEBUG - New I/O worker #260] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.020 [DEBUG - New I/O worker #260] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.021 [DEBUG - New I/O worker #260] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.021 [DEBUG - New I/O worker #260] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.707 [DEBUG - New I/O worker #262] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.049 [DEBUG - New I/O worker #262] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2609), Traces: [0ms] refreshing cache from master, [1472ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [1472ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [1474ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1879ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [2220ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2259ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.050 [DEBUG - New I/O worker #262] (Deferred.java:1330) callback=retry RPC@1622765670 returned Deferred@1686567081(state=PENDING, result=null, callback=(continuation of Deferred@562330205 after retry RPC@1622765670), errback=(continuation of Deferred@562330205 after retry RPC@1622765670)), so the following Deferred is getting paused: Deferred@562330205(state=PAUSED, result=Deferred@1686567081, callback=org.apache.kudu.client.AsyncKuduClient$6@3b7ec5ef -> wakeup thread pool-3-thread-62, errback=passthrough -> wakeup thread pool-3-thread-62) 00:27:35.047 [DEBUG - New I/O worker #217] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.050 [DEBUG - New I/O worker #217] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.050 [DEBUG - New I/O worker #217] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.051 [DEBUG - New I/O worker #217] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.047 [DEBUG - New I/O worker #360] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.051 [DEBUG - New I/O worker #360] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.052 [DEBUG - New I/O worker #360] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.052 [DEBUG - New I/O worker #360] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.052 [DEBUG - New I/O worker #300] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.046 [DEBUG - New I/O worker #275] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.052 [DEBUG - New I/O worker #390] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.053 [DEBUG - New I/O worker #217] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299998, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299998, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299998, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299998, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.054 [DEBUG - pool-3-thread-44] (Deferred.java:1330) callback=disconnect callback@1686011691 returned Deferred@1260951250(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1048101347 after disconnect callback@1686011691), errback=passthrough -> (continuation of Deferred@1048101347 after disconnect callback@1686011691)), so the following Deferred is getting paused: Deferred@1048101347(state=PAUSED, result=Deferred@1260951250, callback=, errback=) 00:27:35.054 [DEBUG - New I/O worker #280] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.055 [DEBUG - New I/O worker #405] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.055 [DEBUG - New I/O worker #217] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.043 [DEBUG - New I/O worker #271] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.021 [DEBUG - New I/O worker #324] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.020 [DEBUG - New I/O worker #108] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.057 [DEBUG - New I/O worker #108] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.058 [DEBUG - New I/O worker #108] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.020 [DEBUG - New I/O worker #252] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.059 [DEBUG - pool-3-thread-32] (Deferred.java:1330) callback=disconnect callback@1056501124 returned Deferred@1354991165(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1716517145 after disconnect callback@1056501124), errback=passthrough -> (continuation of Deferred@1716517145 after disconnect callback@1056501124)), so the following Deferred is getting paused: Deferred@1716517145(state=PAUSED, result=Deferred@1354991165, callback=, errback=) 00:27:35.059 [DEBUG - New I/O worker #180] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.059 [DEBUG - New I/O worker #185] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.019 [DEBUG - New I/O worker #91] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.018 [DEBUG - New I/O worker #28] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.016 [DEBUG - New I/O worker #35] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.060 [DEBUG - New I/O worker #35] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.016 [DEBUG - New I/O worker #351] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.061 [DEBUG - pool-3-thread-12] (Deferred.java:1330) callback=disconnect callback@561085735 returned Deferred@283925503(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1053523625 after disconnect callback@561085735), errback=passthrough -> (continuation of Deferred@1053523625 after disconnect callback@561085735)), so the following Deferred is getting paused: Deferred@1053523625(state=PAUSED, result=Deferred@283925503, callback=, errback=) 00:27:35.061 [DEBUG - New I/O worker #51] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.062 [DEBUG - New I/O worker #93] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.014 [DEBUG - New I/O worker #18] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.064 [DEBUG - New I/O worker #18] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.064 [DEBUG - New I/O worker #18] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.064 [DEBUG - New I/O worker #18] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.012 [DEBUG - New I/O worker #292] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.065 [DEBUG - pool-3-thread-8] (Deferred.java:1330) callback=disconnect callback@1245751631 returned Deferred@1612658803(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@770924748 after disconnect callback@1245751631), errback=passthrough -> (continuation of Deferred@770924748 after disconnect callback@1245751631)), so the following Deferred is getting paused: Deferred@770924748(state=PAUSED, result=Deferred@1612658803, callback=, errback=) 00:27:35.012 [DEBUG - New I/O worker #340] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.066 [DEBUG - New I/O worker #38] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.066 [DEBUG - New I/O worker #58] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.011 [DEBUG - New I/O worker #383] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.011 [DEBUG - pool-3-thread-48] (Deferred.java:1330) callback=disconnect callback@1802203737 returned Deferred@1031824524(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1604810438 after disconnect callback@1802203737), errback=passthrough -> (continuation of Deferred@1604810438 after disconnect callback@1802203737)), so the following Deferred is getting paused: Deferred@1604810438(state=PAUSED, result=Deferred@1031824524, callback=, errback=) 00:27:35.067 [DEBUG - New I/O worker #28] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.010 [DEBUG - New I/O worker #76] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.068 [DEBUG - New I/O worker #76] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.068 [DEBUG - New I/O worker #76] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.068 [DEBUG - New I/O worker #18] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.069 [DEBUG - pool-3-thread-22] (Deferred.java:1330) callback=disconnect callback@376891843 returned Deferred@265294472(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2105373204 after disconnect callback@376891843), errback=passthrough -> (continuation of Deferred@2105373204 after disconnect callback@376891843)), so the following Deferred is getting paused: Deferred@2105373204(state=PAUSED, result=Deferred@265294472, callback=, errback=) 00:27:35.069 [DEBUG - New I/O worker #129] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.069 [DEBUG - New I/O worker #76] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.010 [DEBUG - New I/O worker #253] (Deferred.java:1330) callback=retry RPC@897939095 returned Deferred@834335999(state=PENDING, result=null, callback=(continuation of Deferred@1958247596 after retry RPC@897939095), errback=(continuation of Deferred@1958247596 after retry RPC@897939095)), so the following Deferred is getting paused: Deferred@1958247596(state=PAUSED, result=Deferred@834335999, callback=org.apache.kudu.client.AsyncKuduClient$6@5ba92b53 -> wakeup thread pool-3-thread-65, errback=passthrough -> wakeup thread pool-3-thread-65) 00:27:35.009 [DEBUG - New I/O worker #143] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.071 [DEBUG - New I/O worker #143] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.072 [DEBUG - New I/O worker #143] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.072 [DEBUG - New I/O worker #262] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.072 [DEBUG - New I/O worker #253] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.008 [DEBUG - New I/O worker #528] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.008 [DEBUG - New I/O worker #428] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.008 [DEBUG - pool-3-thread-72] (Deferred.java:1330) callback=disconnect callback@2124354875 returned Deferred@1839169620(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1349469360 after disconnect callback@2124354875), errback=passthrough -> (continuation of Deferred@1349469360 after disconnect callback@2124354875)), so the following Deferred is getting paused: Deferred@1349469360(state=PAUSED, result=Deferred@1839169620, callback=, errback=) 00:27:35.008 [DEBUG - New I/O worker #27] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.075 [DEBUG - New I/O worker #27] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.076 [DEBUG - New I/O worker #27] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.076 [DEBUG - New I/O worker #27] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.076 [DEBUG - New I/O worker #27] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.007 [DEBUG - New I/O worker #71] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.077 [DEBUG - New I/O worker #71] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.077 [DEBUG - New I/O worker #71] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.077 [DEBUG - New I/O worker #83] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.077 [DEBUG - New I/O worker #71] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.078 [DEBUG - New I/O worker #126] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.078 [DEBUG - pool-3-thread-24] (Deferred.java:1330) callback=disconnect callback@487962198 returned Deferred@1151650084(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@528326114 after disconnect callback@487962198), errback=passthrough -> (continuation of Deferred@528326114 after disconnect callback@487962198)), so the following Deferred is getting paused: Deferred@528326114(state=PAUSED, result=Deferred@1151650084, callback=, errback=) 00:27:35.079 [DEBUG - New I/O worker #89] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.079 [DEBUG - New I/O worker #119] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.996 [DEBUG - New I/O worker #103] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.996 [DEBUG - New I/O worker #64] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.995 [DEBUG - New I/O worker #246] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.080 [DEBUG - New I/O worker #246] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.081 [DEBUG - New I/O worker #246] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.993 [DEBUG - New I/O worker #110] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.993 [DEBUG - New I/O worker #92] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.081 [DEBUG - New I/O worker #71] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.993 [DEBUG - New I/O worker #485] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.082 [DEBUG - New I/O worker #323] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.082 [DEBUG - New I/O worker #485] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.082 [DEBUG - New I/O worker #397] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.992 [DEBUG - New I/O worker #138] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.992 [DEBUG - New I/O worker #139] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=2546), Traces: [0ms] refreshing cache from master, [1457ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [1458ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [1458ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [2186ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [2307ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [2390ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:35.083 [DEBUG - New I/O worker #593] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.083 [DEBUG - New I/O worker #139] (Deferred.java:1330) callback=retry RPC@102659796 returned Deferred@35775183(state=PENDING, result=null, callback=(continuation of Deferred@1193508079 after retry RPC@102659796), errback=(continuation of Deferred@1193508079 after retry RPC@102659796)), so the following Deferred is getting paused: Deferred@1193508079(state=PAUSED, result=Deferred@35775183, callback=org.apache.kudu.client.AsyncKuduClient$6@37381c5 -> wakeup thread pool-3-thread-39, errback=passthrough -> wakeup thread pool-3-thread-39) 00:27:34.991 [DEBUG - New I/O worker #184] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.991 [DEBUG - New I/O worker #475] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.084 [DEBUG - New I/O worker #246] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.991 [DEBUG - pool-3-thread-31] (Deferred.java:1330) callback=disconnect callback@308631682 returned Deferred@1771003012(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1850923730 after disconnect callback@308631682), errback=passthrough -> (continuation of Deferred@1850923730 after disconnect callback@308631682)), so the following Deferred is getting paused: Deferred@1850923730(state=PAUSED, result=Deferred@1771003012, callback=, errback=) 00:27:34.989 [DEBUG - New I/O worker #478] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.988 [DEBUG - New I/O worker #597] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.988 [DEBUG - New I/O worker #486] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.987 [DEBUG - New I/O worker #72] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.086 [DEBUG - New I/O worker #72] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.087 [DEBUG - New I/O worker #72] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.088 [DEBUG - New I/O worker #18] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.088 [DEBUG - pool-3-thread-27] (Deferred.java:1330) callback=disconnect callback@414420395 returned Deferred@543584338(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1626784993 after disconnect callback@414420395), errback=passthrough -> (continuation of Deferred@1626784993 after disconnect callback@414420395)), so the following Deferred is getting paused: Deferred@1626784993(state=PAUSED, result=Deferred@543584338, callback=, errback=) 00:27:35.089 [DEBUG - New I/O worker #120] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.089 [DEBUG - New I/O worker #72] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.987 [DEBUG - New I/O worker #115] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.987 [DEBUG - pool-3-thread-28] (Deferred.java:1330) callback=disconnect callback@1899054283 returned Deferred@1659899837(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1559602851 after disconnect callback@1899054283), errback=passthrough -> (continuation of Deferred@1559602851 after disconnect callback@1899054283)), so the following Deferred is getting paused: Deferred@1559602851(state=PAUSED, result=Deferred@1659899837, callback=, errback=) 00:27:34.985 [DEBUG - New I/O worker #11] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.090 [DEBUG - New I/O worker #11] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.984 [DEBUG - New I/O worker #333] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.983 [DEBUG - New I/O worker #268] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.983 [DEBUG - New I/O worker #218] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.981 [DEBUG - New I/O worker #257] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.092 [DEBUG - New I/O worker #139] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.092 [DEBUG - New I/O worker #185] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.092 [DEBUG - New I/O worker #23] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.093 [DEBUG - New I/O worker #37] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.093 [DEBUG - New I/O worker #252] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.093 [DEBUG - New I/O worker #11] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.094 [DEBUG - New I/O worker #217] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.954 [DEBUG - New I/O worker #415] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.954 [DEBUG - New I/O worker #413] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.947 [DEBUG - New I/O worker #100] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.946 [DEBUG - New I/O worker #472] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:34.944 [DEBUG - New I/O worker #19] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.143 [DEBUG - New I/O worker #472] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.944 [DEBUG - New I/O worker #137] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.944 [DEBUG - New I/O worker #305] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.944 [DEBUG - New I/O worker #264] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.144 [DEBUG - New I/O worker #472] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.145 [DEBUG - New I/O worker #472] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.943 [DEBUG - New I/O worker #70] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.942 [DEBUG - New I/O worker #142] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.942 [DEBUG - New I/O worker #118] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.942 [DEBUG - New I/O worker #90] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.942 [DEBUG - New I/O worker #134] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.146 [DEBUG - New I/O worker #490] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.146 [DEBUG - New I/O worker #588] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.940 [DEBUG - pool-3-thread-37] (Deferred.java:1330) callback=disconnect callback@653692245 returned Deferred@64871256(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@669275300 after disconnect callback@653692245), errback=passthrough -> (continuation of Deferred@669275300 after disconnect callback@653692245)), so the following Deferred is getting paused: Deferred@669275300(state=PAUSED, result=Deferred@64871256, callback=, errback=) 00:27:34.939 [DEBUG - pool-3-thread-9] (Deferred.java:1330) callback=disconnect callback@1246280405 returned Deferred@75650492(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1287133392 after disconnect callback@1246280405), errback=passthrough -> (continuation of Deferred@1287133392 after disconnect callback@1246280405)), so the following Deferred is getting paused: Deferred@1287133392(state=PAUSED, result=Deferred@75650492, callback=, errback=) 00:27:34.938 [DEBUG - New I/O worker #517] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.938 [DEBUG - New I/O worker #30] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.937 [DEBUG - New I/O worker #396] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.149 [DEBUG - New I/O worker #139] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.149 [DEBUG - New I/O worker #69] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.935 [DEBUG - New I/O worker #266] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.918 [DEBUG - New I/O worker #322] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.917 [DEBUG - New I/O worker #220] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.151 [DEBUG - New I/O worker #472] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.917 [DEBUG - New I/O worker #128] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.891 [DEBUG - New I/O worker #13] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.891 [DEBUG - New I/O worker #291] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:34.888 [DEBUG - New I/O worker #277] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.153 [DEBUG - New I/O worker #277] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.153 [DEBUG - New I/O worker #277] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.153 [DEBUG - New I/O worker #277] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.154 [DEBUG - New I/O worker #277] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.885 [DEBUG - New I/O worker #493] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.154 [DEBUG - New I/O worker #493] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.155 [DEBUG - New I/O worker #493] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.157 [DEBUG - New I/O worker #493] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.885 [DEBUG - New I/O worker #22] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.883 [DEBUG - New I/O worker #102] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.883 [DEBUG - New I/O worker #105] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.883 [DEBUG - New I/O worker #219] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.180 [DEBUG - pool-3-thread-7] (Deferred.java:1330) callback=disconnect callback@885627272 returned Deferred@234309326(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@621065679 after disconnect callback@885627272), errback=passthrough -> (continuation of Deferred@621065679 after disconnect callback@885627272)), so the following Deferred is getting paused: Deferred@621065679(state=PAUSED, result=Deferred@234309326, callback=, errback=) 00:27:35.181 [DEBUG - New I/O worker #54] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.883 [DEBUG - New I/O worker #337] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.181 [DEBUG - New I/O worker #56] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.883 [DEBUG - New I/O worker #259] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.883 [DEBUG - New I/O worker #107] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.882 [DEBUG - New I/O worker #258] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.882 [DEBUG - pool-3-thread-45] (Deferred.java:1330) callback=disconnect callback@1011150462 returned Deferred@484868390(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1534619823 after disconnect callback@1011150462), errback=passthrough -> (continuation of Deferred@1534619823 after disconnect callback@1011150462)), so the following Deferred is getting paused: Deferred@1534619823(state=PAUSED, result=Deferred@484868390, callback=, errback=) 00:27:34.882 [DEBUG - New I/O worker #14] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.882 [DEBUG - New I/O worker #12] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.882 [DEBUG - New I/O worker #41] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.882 [DEBUG - New I/O worker #301] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.882 [DEBUG - New I/O worker #221] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.882 [DEBUG - New I/O worker #269] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.880 [DEBUG - New I/O worker #350] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.880 [DEBUG - New I/O worker #116] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.880 [DEBUG - New I/O worker #183] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.880 [DEBUG - New I/O worker #247] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.880 [DEBUG - New I/O worker #241] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.880 [DEBUG - New I/O worker #352] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.880 [DEBUG - New I/O worker #24] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.880 [DEBUG - New I/O worker #114] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.880 [DEBUG - New I/O worker #112] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.880 [DEBUG - New I/O worker #193] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.879 [DEBUG - pool-3-thread-59] (Deferred.java:1330) callback=disconnect callback@2108966594 returned Deferred@2133161484(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@290614553 after disconnect callback@2108966594), errback=passthrough -> (continuation of Deferred@290614553 after disconnect callback@2108966594)), so the following Deferred is getting paused: Deferred@290614553(state=PAUSED, result=Deferred@2133161484, callback=, errback=) 00:27:34.879 [DEBUG - pool-3-thread-34] (Deferred.java:1330) callback=disconnect callback@2121169543 returned Deferred@923825205(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@899735025 after disconnect callback@2121169543), errback=passthrough -> (continuation of Deferred@899735025 after disconnect callback@2121169543)), so the following Deferred is getting paused: Deferred@899735025(state=PAUSED, result=Deferred@923825205, callback=, errback=) 00:27:34.879 [DEBUG - New I/O worker #304] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.879 [DEBUG - New I/O worker #416] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.879 [DEBUG - New I/O worker #425] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.879 [DEBUG - New I/O worker #494] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.879 [DEBUG - New I/O worker #141] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.878 [DEBUG - New I/O worker #492] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.878 [DEBUG - New I/O worker #29] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.878 [DEBUG - New I/O worker #376] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.878 [DEBUG - New I/O worker #43] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.878 [DEBUG - New I/O worker #77] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.878 [DEBUG - New I/O worker #389] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.878 [DEBUG - New I/O worker #365] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.878 [DEBUG - New I/O worker #354] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.878 [DEBUG - New I/O worker #17] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.878 [DEBUG - New I/O worker #49] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.878 [DEBUG - New I/O worker #402] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.878 [DEBUG - New I/O worker #434] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.878 [DEBUG - New I/O worker #274] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.878 [DEBUG - New I/O worker #50] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.878 [DEBUG - New I/O worker #80] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.878 [DEBUG - New I/O worker #79] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.876 [DEBUG - New I/O worker #86] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.875 [DEBUG - New I/O worker #265] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.873 [DEBUG - New I/O worker #78] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:34.869 [DEBUG - New I/O worker #122] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.869 [DEBUG - New I/O worker #75] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:34.869 [DEBUG - New I/O worker #476] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.869 [DEBUG - New I/O worker #127] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:34.869 [DEBUG - New I/O worker #375] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.868 [DEBUG - New I/O worker #270] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.868 [DEBUG - New I/O worker #10] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:34.834 [DEBUG - New I/O worker #67] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.197 [DEBUG - New I/O worker #67] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:34.832 [DEBUG - New I/O worker #53] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:34.832 [DEBUG - New I/O worker #442] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.198 [DEBUG - pool-3-thread-25] (Deferred.java:1330) callback=disconnect callback@1324933132 returned Deferred@1507786255(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1990292012 after disconnect callback@1324933132), errback=passthrough -> (continuation of Deferred@1990292012 after disconnect callback@1324933132)), so the following Deferred is getting paused: Deferred@1990292012(state=PAUSED, result=Deferred@1507786255, callback=, errback=) 00:27:35.199 [DEBUG - New I/O worker #109] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.199 [DEBUG - New I/O worker #111] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.200 [DEBUG - New I/O worker #78] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.201 [DEBUG - New I/O worker #78] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.179 [DEBUG - New I/O worker #277] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.202 [DEBUG - New I/O worker #85] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.202 [DEBUG - New I/O worker #124] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.172 [DEBUG - New I/O worker #305] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.171 [DEBUG - New I/O worker #70] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.169 [DEBUG - New I/O worker #100] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.168 [DEBUG - New I/O worker #19] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.168 [DEBUG - New I/O worker #137] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.205 [DEBUG - New I/O worker #22] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.167 [DEBUG - New I/O worker #142] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.167 [DEBUG - New I/O worker #266] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.156 [DEBUG - New I/O worker #482] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.156 [DEBUG - pool-3-thread-51] (Deferred.java:1330) callback=disconnect callback@455193761 returned Deferred@766308878(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1245617635 after disconnect callback@455193761), errback=passthrough -> (continuation of Deferred@1245617635 after disconnect callback@455193761)), so the following Deferred is getting paused: Deferred@1245617635(state=PAUSED, result=Deferred@766308878, callback=, errback=) 00:27:35.156 [DEBUG - New I/O worker #359] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.154 [DEBUG - New I/O worker #11] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.153 [DEBUG - New I/O worker #291] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.207 [DEBUG - New I/O worker #291] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.207 [DEBUG - New I/O worker #291] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.208 [DEBUG - New I/O worker #291] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.150 [DEBUG - New I/O worker #30] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.150 [DEBUG - New I/O worker #45] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.149 [DEBUG - New I/O worker #139] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.209 [DEBUG - New I/O worker #139] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.209 [DEBUG - New I/O worker #139] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.209 [DEBUG - New I/O worker #379] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.149 [DEBUG - pool-3-thread-13] (Deferred.java:1330) callback=disconnect callback@1845392686 returned Deferred@1984180240(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@899766680 after disconnect callback@1845392686), errback=passthrough -> (continuation of Deferred@899766680 after disconnect callback@1845392686)), so the following Deferred is getting paused: Deferred@899766680(state=PAUSED, result=Deferred@1984180240, callback=, errback=) 00:27:35.146 [DEBUG - New I/O worker #134] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.211 [DEBUG - New I/O worker #134] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.212 [DEBUG - pool-3-thread-40] (Deferred.java:1330) callback=disconnect callback@1166941881 returned Deferred@815405252(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@410851911 after disconnect callback@1166941881), errback=passthrough -> (continuation of Deferred@410851911 after disconnect callback@1166941881)), so the following Deferred is getting paused: Deferred@410851911(state=PAUSED, result=Deferred@815405252, callback=, errback=) 00:27:35.212 [DEBUG - New I/O worker #175] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.146 [DEBUG - pool-3-thread-78] (Deferred.java:1330) callback=disconnect callback@223460553 returned Deferred@1415076189(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@560451029 after disconnect callback@223460553), errback=passthrough -> (continuation of Deferred@560451029 after disconnect callback@223460553)), so the following Deferred is getting paused: Deferred@560451029(state=PAUSED, result=Deferred@1415076189, callback=, errback=) 00:27:35.144 [DEBUG - New I/O worker #264] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.214 [DEBUG - New I/O worker #264] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.214 [DEBUG - New I/O worker #264] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.214 [DEBUG - New I/O worker #264] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.214 [DEBUG - New I/O worker #219] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.215 [DEBUG - pool-3-thread-61] (Deferred.java:1330) callback=disconnect callback@1888591633 returned Deferred@1502457414(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1552966664 after disconnect callback@1888591633), errback=passthrough -> (continuation of Deferred@1552966664 after disconnect callback@1888591633)), so the following Deferred is getting paused: Deferred@1552966664(state=PAUSED, result=Deferred@1502457414, callback=, errback=) 00:27:35.216 [DEBUG - New I/O worker #349] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.140 [DEBUG - New I/O worker #138] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.216 [DEBUG - New I/O worker #373] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.217 [DEBUG - New I/O worker #105] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.135 [DEBUG - New I/O worker #333] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.123 [DEBUG - New I/O worker #292] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.120 [DEBUG - New I/O worker #76] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.218 [DEBUG - New I/O worker #586] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.218 [DEBUG - New I/O worker #660] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.114 [DEBUG - New I/O worker #110] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.113 [DEBUG - New I/O worker #71] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.113 [DEBUG - New I/O worker #246] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.112 [DEBUG - New I/O worker #478] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.112 [DEBUG - New I/O worker #72] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.111 [DEBUG - New I/O worker #268] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.221 [DEBUG - New I/O worker #493] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.108 [DEBUG - New I/O worker #262] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.248 [DEBUG - New I/O worker #262] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.249 [DEBUG - New I/O worker #262] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.249 [DEBUG - New I/O worker #262] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.249 [DEBUG - New I/O worker #262] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.092 [DEBUG - pool-3-thread-3] (Deferred.java:1330) callback=disconnect callback@1063601715 returned Deferred@1789707049(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2006604020 after disconnect callback@1063601715), errback=passthrough -> (continuation of Deferred@2006604020 after disconnect callback@1063601715)), so the following Deferred is getting paused: Deferred@2006604020(state=PAUSED, result=Deferred@1789707049, callback=, errback=) 00:27:35.251 [DEBUG - pool-3-thread-62] (Deferred.java:1330) callback=disconnect callback@2138427894 returned Deferred@186643331(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@383874738 after disconnect callback@2138427894), errback=passthrough -> (continuation of Deferred@383874738 after disconnect callback@2138427894)), so the following Deferred is getting paused: Deferred@383874738(state=PAUSED, result=Deferred@186643331, callback=, errback=) 00:27:35.092 [DEBUG - New I/O worker #257] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.091 [DEBUG - New I/O worker #91] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.254 [DEBUG - pool-3-thread-64] (Deferred.java:1330) callback=disconnect callback@6654656 returned Deferred@1175469123(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1526965598 after disconnect callback@6654656), errback=passthrough -> (continuation of Deferred@1526965598 after disconnect callback@6654656)), so the following Deferred is getting paused: Deferred@1526965598(state=PAUSED, result=Deferred@1175469123, callback=, errback=) 00:27:35.088 [DEBUG - New I/O worker #130] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.254 [DEBUG - New I/O worker #328] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.255 [DEBUG - New I/O worker #388] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.084 [DEBUG - New I/O worker #485] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.084 [DEBUG - New I/O worker #475] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.084 [DEBUG - New I/O worker #631] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.083 [DEBUG - pool-3-thread-80] (Deferred.java:1330) callback=disconnect callback@1768381885 returned Deferred@1965614216(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@915310102 after disconnect callback@1768381885), errback=passthrough -> (continuation of Deferred@915310102 after disconnect callback@1768381885)), so the following Deferred is getting paused: Deferred@915310102(state=PAUSED, result=Deferred@1965614216, callback=, errback=) 00:27:35.082 [DEBUG - pool-3-thread-69] (Deferred.java:1330) callback=disconnect callback@1510160942 returned Deferred@1326036165(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1163494371 after disconnect callback@1510160942), errback=passthrough -> (continuation of Deferred@1163494371 after disconnect callback@1510160942)), so the following Deferred is getting paused: Deferred@1163494371(state=PAUSED, result=Deferred@1326036165, callback=, errback=) 00:27:35.079 [DEBUG - New I/O worker #27] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.077 [DEBUG - pool-3-thread-11] (Deferred.java:1330) callback=disconnect callback@1942617930 returned Deferred@201039133(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1789576904 after disconnect callback@1942617930), errback=passthrough -> (continuation of Deferred@1789576904 after disconnect callback@1942617930)), so the following Deferred is getting paused: Deferred@1789576904(state=PAUSED, result=Deferred@201039133, callback=, errback=) 00:27:35.076 [DEBUG - New I/O worker #253] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.260 [DEBUG - New I/O worker #253] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.260 [DEBUG - New I/O worker #253] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.261 [DEBUG - New I/O worker #253] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.261 [DEBUG - New I/O worker #253] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.072 [DEBUG - New I/O worker #143] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.069 [DEBUG - New I/O worker #121] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.263 [DEBUG - pool-3-thread-42] (Deferred.java:1330) callback=disconnect callback@392033973 returned Deferred@1365042420(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1647335996 after disconnect callback@392033973), errback=passthrough -> (continuation of Deferred@1647335996 after disconnect callback@392033973)), so the following Deferred is getting paused: Deferred@1647335996(state=PAUSED, result=Deferred@1365042420, callback=, errback=) 00:27:35.067 [DEBUG - New I/O worker #35] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.063 [DEBUG - New I/O worker #390] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.263 [DEBUG - New I/O worker #289] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.059 [DEBUG - New I/O worker #108] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.264 [DEBUG - New I/O worker #418] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.053 [DEBUG - New I/O worker #275] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.264 [DEBUG - New I/O worker #275] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.264 [DEBUG - New I/O worker #275] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.265 [DEBUG - New I/O worker #275] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.052 [DEBUG - New I/O worker #360] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.266 [DEBUG - pool-3-thread-56] (Deferred.java:1330) callback=disconnect callback@796384424 returned Deferred@475498159(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@823239540 after disconnect callback@796384424), errback=passthrough -> (continuation of Deferred@823239540 after disconnect callback@796384424)), so the following Deferred is getting paused: Deferred@823239540(state=PAUSED, result=Deferred@475498159, callback=, errback=) 00:27:35.052 [DEBUG - New I/O worker #260] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.266 [DEBUG - New I/O worker #143] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.267 [DEBUG - New I/O worker #348] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.267 [DEBUG - New I/O worker #374] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.052 [DEBUG - pool-3-thread-63] (Deferred.java:1330) callback=disconnect callback@1880351530 returned Deferred@498389797(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1346615782 after disconnect callback@1880351530), errback=passthrough -> (continuation of Deferred@1346615782 after disconnect callback@1880351530)), so the following Deferred is getting paused: Deferred@1346615782(state=PAUSED, result=Deferred@498389797, callback=, errback=) 00:27:35.269 [DEBUG - New I/O worker #360] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.270 [DEBUG - pool-3-thread-74] (Deferred.java:1330) callback=disconnect callback@505858076 returned Deferred@1473963075(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1247982727 after disconnect callback@505858076), errback=passthrough -> (continuation of Deferred@1247982727 after disconnect callback@505858076)), so the following Deferred is getting paused: Deferred@1247982727(state=PAUSED, result=Deferred@1473963075, callback=, errback=) 00:27:35.270 [DEBUG - New I/O worker #481] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.270 [DEBUG - New I/O worker #483] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.260 [DEBUG - New I/O worker #475] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.272 [DEBUG - pool-3-thread-65] (Deferred.java:1330) callback=disconnect callback@1309124845 returned Deferred@815235928(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2118142414 after disconnect callback@1309124845), errback=passthrough -> (continuation of Deferred@2118142414 after disconnect callback@1309124845)), so the following Deferred is getting paused: Deferred@2118142414(state=PAUSED, result=Deferred@815235928, callback=, errback=) 00:27:35.272 [DEBUG - New I/O worker #356] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.272 [DEBUG - pool-3-thread-77] (Deferred.java:1330) callback=disconnect callback@274513988 returned Deferred@640327645(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@386395975 after disconnect callback@274513988), errback=passthrough -> (continuation of Deferred@386395975 after disconnect callback@274513988)), so the following Deferred is getting paused: Deferred@386395975(state=PAUSED, result=Deferred@640327645, callback=, errback=) 00:27:35.272 [DEBUG - New I/O worker #592] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.273 [DEBUG - New I/O worker #326] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.274 [DEBUG - New I/O worker #253] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.275 [DEBUG - New I/O worker #475] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.276 [DEBUG - New I/O worker #275] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.259 [DEBUG - New I/O worker #262] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.259 [DEBUG - New I/O worker #257] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.251 [DEBUG - New I/O worker #398] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.251 [DEBUG - New I/O worker #299] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.245 [DEBUG - New I/O worker #134] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.245 [DEBUG - New I/O worker #264] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.245 [DEBUG - New I/O worker #291] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.242 [DEBUG - New I/O worker #277] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.239 [DEBUG - New I/O worker #476] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.239 [DEBUG - New I/O worker #79] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.239 [DEBUG - New I/O worker #274] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.239 [DEBUG - New I/O worker #17] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.238 [DEBUG - New I/O worker #77] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.238 [DEBUG - New I/O worker #247] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.237 [DEBUG - New I/O worker #269] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.237 [DEBUG - New I/O worker #259] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.237 [DEBUG - New I/O worker #102] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.237 [DEBUG - New I/O worker #14] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.236 [DEBUG - New I/O worker #12] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.236 [DEBUG - New I/O worker #116] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.234 [DEBUG - New I/O worker #114] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.234 [DEBUG - New I/O worker #416] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.234 [DEBUG - New I/O worker #494] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.233 [DEBUG - New I/O worker #141] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.233 [DEBUG - New I/O worker #29] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.233 [DEBUG - New I/O worker #376] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.232 [DEBUG - New I/O worker #43] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.232 [DEBUG - New I/O worker #389] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.232 [DEBUG - New I/O worker #365] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.232 [DEBUG - New I/O worker #49] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.231 [DEBUG - New I/O worker #402] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.231 [DEBUG - New I/O worker #122] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.231 [DEBUG - New I/O worker #127] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.227 [DEBUG - New I/O worker #22] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.225 [DEBUG - New I/O worker #30] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.220 [DEBUG - New I/O worker #472] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.218 [DEBUG - Time-limited test] (Deferred.java:1330) callback=disconnect callback@1571647994 returned Deferred@1931429039(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1817408491 after disconnect callback@1571647994), errback=passthrough -> (continuation of Deferred@1817408491 after disconnect callback@1571647994)), so the following Deferred is getting paused: Deferred@1817408491(state=PAUSED, result=Deferred@1931429039, callback=, errback=) 00:27:35.213 [DEBUG - New I/O worker #178] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.213 [DEBUG - New I/O worker #78] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.210 [DEBUG - New I/O worker #139] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.209 [DEBUG - New I/O worker #344] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.304 [DEBUG - pool-3-thread-39] (Deferred.java:1330) callback=disconnect callback@1965742251 returned Deferred@747061648(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1190537825 after disconnect callback@1965742251), errback=passthrough -> (continuation of Deferred@1190537825 after disconnect callback@1965742251)), so the following Deferred is getting paused: Deferred@1190537825(state=PAUSED, result=Deferred@747061648, callback=, errback=) 00:27:35.209 [DEBUG - pool-3-thread-46] (Deferred.java:1330) callback=disconnect callback@2023501344 returned Deferred@988710167(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1411335087 after disconnect callback@2023501344), errback=passthrough -> (continuation of Deferred@1411335087 after disconnect callback@2023501344)), so the following Deferred is getting paused: Deferred@1411335087(state=PAUSED, result=Deferred@988710167, callback=, errback=) 00:27:35.202 [DEBUG - pool-3-thread-20] (Deferred.java:1330) callback=disconnect callback@180595996 returned Deferred@568947989(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1608403638 after disconnect callback@180595996), errback=passthrough -> (continuation of Deferred@1608403638 after disconnect callback@180595996)), so the following Deferred is getting paused: Deferred@1608403638(state=PAUSED, result=Deferred@568947989, callback=, errback=) 00:27:35.201 [DEBUG - New I/O worker #67] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.304 [DEBUG - New I/O worker #144] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.281 [DEBUG - New I/O worker #493] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.273 [DEBUG - New I/O worker #637] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.308 [DEBUG - New I/O worker #264] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.307 [DEBUG - New I/O worker #291] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.304 [DEBUG - New I/O worker #139] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.304 [DEBUG - New I/O worker #288] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.319 [DEBUG - New I/O worker #134] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.320 [DEBUG - New I/O worker #257] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.322 [DEBUG - New I/O worker #262] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.322 [DEBUG - New I/O worker #275] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.323 [DEBUG - New I/O worker #475] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.323 [DEBUG - New I/O worker #253] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.324 [DEBUG - New I/O worker #483] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.325 [DEBUG - New I/O worker #143] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.327 [DEBUG - New I/O worker #35] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.330 [DEBUG - New I/O worker #485] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.331 [DEBUG - New I/O worker #27] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.354 [DEBUG - New I/O worker #67] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.366 [DEBUG - New I/O worker #78] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.368 [DEBUG - pool-3-thread-2] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.386 [DEBUG - New I/O worker #805] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.387 [DEBUG - New I/O worker #804] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.388 [DEBUG - pool-3-thread-45] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.389 [DEBUG - New I/O worker #806] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.409 [DEBUG - New I/O worker #805] (Negotiator.java:764) Authenticated connection [id: 0xdc96c144, /127.0.0.1:53058 => /127.22.19.254:44849] using SASL/PLAIN 00:27:35.420 [DEBUG - New I/O worker #806] (Negotiator.java:764) Authenticated connection [id: 0xf3d6f22a, /127.0.0.1:39688 => /127.22.19.253:37879] using SASL/PLAIN 00:27:35.428 [DEBUG - New I/O worker #804] (Negotiator.java:764) Authenticated connection [id: 0xad4da712, /127.0.0.1:37712 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.432 [DEBUG - New I/O worker #804] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.435 [DEBUG - New I/O worker #804] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=66), Traces: [0ms] refreshing cache from master, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [12ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [13ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [56ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [56ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [63ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.438 [DEBUG - New I/O worker #804] (Deferred.java:1330) callback=retry RPC@373951153 returned Deferred@309693600(state=PENDING, result=null, callback=(continuation of Deferred@1467416714 after retry RPC@373951153), errback=(continuation of Deferred@1467416714 after retry RPC@373951153)), so the following Deferred is getting paused: Deferred@1467416714(state=PAUSED, result=Deferred@309693600, callback=org.apache.kudu.client.AsyncKuduClient$6@2033ded3 -> wakeup thread pool-3-thread-2, errback=passthrough -> wakeup thread pool-3-thread-2) 00:27:35.440 [DEBUG - New I/O worker #804] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.445 [DEBUG - New I/O worker #804] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.445 [DEBUG - New I/O worker #804] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.445 [DEBUG - New I/O worker #804] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.446 [DEBUG - pool-3-thread-43] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.446 [DEBUG - New I/O worker #804] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.446 [DEBUG - New I/O worker #804] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.447 [DEBUG - pool-3-thread-2] (Deferred.java:1330) callback=disconnect callback@148125863 returned Deferred@324478056(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1035795063 after disconnect callback@148125863), errback=passthrough -> (continuation of Deferred@1035795063 after disconnect callback@148125863)), so the following Deferred is getting paused: Deferred@1035795063(state=PAUSED, result=Deferred@324478056, callback=, errback=) 00:27:35.447 [DEBUG - New I/O worker #806] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.449 [DEBUG - New I/O worker #805] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.449 [DEBUG - New I/O worker #804] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.482 [DEBUG - New I/O worker #139] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.517 [DEBUG - pool-3-thread-26] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.531 [DEBUG - New I/O worker #907] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.531 [DEBUG - New I/O worker #771] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.532 [DEBUG - New I/O worker #921] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.531 [DEBUG - New I/O worker #920] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.546 [DEBUG - New I/O worker #921] (Negotiator.java:764) Authenticated connection [id: 0x320891a3, /127.0.0.1:39694 => /127.22.19.253:37879] using SASL/PLAIN 00:27:35.547 [DEBUG - New I/O worker #907] (Negotiator.java:764) Authenticated connection [id: 0x55bfd0f3, /127.0.0.1:39700 => /127.22.19.253:37879] using SASL/PLAIN 00:27:35.548 [DEBUG - New I/O worker #771] (Negotiator.java:764) Authenticated connection [id: 0xab0a7b9b, /127.0.0.1:37718 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.551 [DEBUG - pool-3-thread-69] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.552 [DEBUG - pool-3-thread-58] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.552 [DEBUG - New I/O worker #771] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.553 [DEBUG - New I/O worker #771] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=107), Traces: [0ms] refreshing cache from master, [68ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [68ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [69ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [106ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.556 [DEBUG - New I/O worker #814] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.556 [DEBUG - New I/O worker #771] (Deferred.java:1330) callback=retry RPC@1544680516 returned Deferred@1479441182(state=PENDING, result=null, callback=(continuation of Deferred@489487999 after retry RPC@1544680516), errback=(continuation of Deferred@489487999 after retry RPC@1544680516)), so the following Deferred is getting paused: Deferred@489487999(state=PAUSED, result=Deferred@1479441182, callback=org.apache.kudu.client.AsyncKuduClient$6@7f1bc8d4 -> wakeup thread pool-3-thread-43, errback=passthrough -> wakeup thread pool-3-thread-43) 00:27:35.556 [DEBUG - New I/O worker #771] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.560 [DEBUG - New I/O worker #771] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.561 [DEBUG - New I/O worker #771] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.561 [DEBUG - New I/O worker #771] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.561 [DEBUG - New I/O worker #771] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.561 [DEBUG - New I/O worker #771] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.562 [DEBUG - pool-3-thread-43] (Deferred.java:1330) callback=disconnect callback@2025302684 returned Deferred@1660986098(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1197247326 after disconnect callback@2025302684), errback=passthrough -> (continuation of Deferred@1197247326 after disconnect callback@2025302684)), so the following Deferred is getting paused: Deferred@1197247326(state=PAUSED, result=Deferred@1660986098, callback=, errback=) 00:27:35.562 [DEBUG - New I/O worker #921] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.563 [DEBUG - New I/O worker #873] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.564 [DEBUG - New I/O worker #771] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.569 [DEBUG - New I/O worker #779] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.570 [DEBUG - New I/O worker #864] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.571 [DEBUG - pool-3-thread-21] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:35.573113 24498 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.515665 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53064 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.516160 (+ 495us) server_negotiation.cc:184] Beginning negotiation 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.516183 (+ 23us) server_negotiation.cc:373] Waiting for connection header 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.517282 (+ 1099us) server_negotiation.cc:381] Connection header received 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.519336 (+ 2054us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.519391 (+ 55us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.519531 (+ 140us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.519669 (+ 138us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.544003 (+ 24334us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.544873 (+ 870us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.554953 (+ 10080us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:35.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.556217 (+ 1264us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:35.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.556400 (+ 183us) server_negotiation.cc:589] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 00:27:35.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.557431 (+ 1031us) server_negotiation.cc:337] Received SASL_INITIATE NegotiatePB request 00:27:35.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.557457 (+ 26us) server_negotiation.cc:746] Received SASL_INITIATE request from client 00:27:35.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.557484 (+ 27us) server_negotiation.cc:757] Client requested to use mechanism: PLAIN 00:27:35.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.557535 (+ 51us) server_negotiation.cc:786] Calling sasl_server_start() 00:27:35.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.557589 (+ 54us) server_negotiation.cc:945] Received PLAIN auth, user=slave 00:27:35.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.557627 (+ 38us) server_negotiation.cc:349] Sending SASL_SUCCESS NegotiatePB response 00:27:35.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.557888 (+ 261us) server_negotiation.cc:890] Waiting for connection context 00:27:35.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.571892 (+ 14004us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53064: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:53064): Cannot send after transport endpoint shutdown (error 108) 00:27:35.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":147} 00:27:35.574 [DEBUG - New I/O worker #920] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.577 [DEBUG - New I/O worker #778] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.582 [DEBUG - New I/O worker #814] (Negotiator.java:764) Authenticated connection [id: 0x11e5eb70, /127.0.0.1:37730 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.583 [DEBUG - New I/O worker #779] (Negotiator.java:764) Authenticated connection [id: 0x4a16e762, /127.0.0.1:53070 => /127.22.19.254:44849] using SASL/PLAIN 00:27:35.585 [DEBUG - New I/O worker #814] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.586 [DEBUG - New I/O worker #814] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=197), Traces: [0ms] refreshing cache from master, [164ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [197ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.589 [DEBUG - New I/O worker #814] (Deferred.java:1330) callback=retry RPC@2084478486 returned Deferred@2013336905(state=PENDING, result=null, callback=(continuation of Deferred@1023644717 after retry RPC@2084478486), errback=(continuation of Deferred@1023644717 after retry RPC@2084478486)), so the following Deferred is getting paused: Deferred@1023644717(state=PAUSED, result=Deferred@2013336905, callback=org.apache.kudu.client.AsyncKuduClient$6@4322d02 -> wakeup thread pool-3-thread-45, errback=passthrough -> wakeup thread pool-3-thread-45) 00:27:35.589 [DEBUG - New I/O worker #814] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.590 [DEBUG - New I/O worker #873] (Negotiator.java:764) Authenticated connection [id: 0x0884fc13, /127.0.0.1:39706 => /127.22.19.253:37879] using SASL/PLAIN 00:27:35.593 [DEBUG - New I/O worker #778] (Negotiator.java:764) Authenticated connection [id: 0x02136ceb, /127.0.0.1:37724 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.595 [DEBUG - New I/O worker #814] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.595 [DEBUG - New I/O worker #814] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.595 [DEBUG - New I/O worker #814] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.595 [DEBUG - New I/O worker #814] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.596 [DEBUG - New I/O worker #814] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.596 [DEBUG - New I/O worker #778] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.596 [DEBUG - New I/O worker #778] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=79), Traces: [0ms] refreshing cache from master, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [45ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [70ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [78ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.597 [DEBUG - New I/O worker #814] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.599 [DEBUG - New I/O worker #778] (Deferred.java:1330) callback=retry RPC@1870724577 returned Deferred@1807709385(state=PENDING, result=null, callback=(continuation of Deferred@784201690 after retry RPC@1870724577), errback=(continuation of Deferred@784201690 after retry RPC@1870724577)), so the following Deferred is getting paused: Deferred@784201690(state=PAUSED, result=Deferred@1807709385, callback=org.apache.kudu.client.AsyncKuduClient$6@77de9f63 -> wakeup thread pool-3-thread-26, errback=passthrough -> wakeup thread pool-3-thread-26) 00:27:35.599 [DEBUG - New I/O worker #778] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:35.600018 24599 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.553766 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53076 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.569253 (+ 15487us) server_negotiation.cc:184] Beginning negotiation 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.569287 (+ 34us) server_negotiation.cc:373] Waiting for connection header 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.569327 (+ 40us) server_negotiation.cc:381] Connection header received 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.569429 (+ 102us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.569456 (+ 27us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.569584 (+ 128us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.571053 (+ 1469us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.576767 (+ 5714us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.577915 (+ 1148us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.596362 (+ 18447us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.598052 (+ 1690us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.598535 (+ 483us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53076: BlockingWrite error: write error: Broken pipe (error 32) 00:27:35.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":15191,"thread_start_us":14057,"threads_started":1} 00:27:35.603 [DEBUG - New I/O worker #778] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.603 [DEBUG - New I/O worker #778] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.603 [DEBUG - New I/O worker #778] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.604 [DEBUG - New I/O worker #778] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.604 [DEBUG - New I/O worker #778] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.605 [DEBUG - pool-3-thread-26] (Deferred.java:1330) callback=disconnect callback@1905777951 returned Deferred@410832144(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@980156856 after disconnect callback@1905777951), errback=passthrough -> (continuation of Deferred@980156856 after disconnect callback@1905777951)), so the following Deferred is getting paused: Deferred@980156856(state=PAUSED, result=Deferred@410832144, callback=, errback=) 00:27:35.605 [DEBUG - New I/O worker #907] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.605 [DEBUG - New I/O worker #779] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.606 [DEBUG - New I/O worker #778] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.607 [DEBUG - New I/O worker #778] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.665 [DEBUG - pool-3-thread-13] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.720 [DEBUG - New I/O worker #864] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:35.720 [DEBUG - pool-3-thread-45] (Deferred.java:1330) callback=disconnect callback@1389714845 returned Deferred@736318051(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1560327050 after disconnect callback@1389714845), errback=passthrough -> (continuation of Deferred@1560327050 after disconnect callback@1389714845)), so the following Deferred is getting paused: Deferred@1560327050(state=PAUSED, result=Deferred@736318051, callback=, errback=) 00:27:35.720 [DEBUG - New I/O worker #804] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.721 [DEBUG - New I/O worker #873] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.720 [INFO - New I/O worker #864] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:35.722 [DEBUG - New I/O worker #774] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.723 [DEBUG - New I/O worker #788] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.722 [DEBUG - New I/O worker #831] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.723 [DEBUG - New I/O worker #777] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.724 [DEBUG - New I/O worker #895] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.731 [DEBUG - New I/O worker #896] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.734 [DEBUG - New I/O worker #830] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.734 [DEBUG - New I/O worker #829] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.737 [DEBUG - New I/O worker #889] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.738 [DEBUG - New I/O worker #788] (Negotiator.java:764) Authenticated connection [id: 0x044b732d, /127.0.0.1:37742 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.743 [DEBUG - New I/O worker #896] (Negotiator.java:764) Authenticated connection [id: 0x12c0dc72, /127.0.0.1:39718 => /127.22.19.253:37879] using SASL/PLAIN 00:27:35.744 [DEBUG - pool-3-thread-10] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.745 [DEBUG - pool-3-thread-30] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.745 [DEBUG - pool-3-thread-4] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.746 [DEBUG - pool-3-thread-78] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.748 [DEBUG - pool-3-thread-28] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.748 [DEBUG - pool-3-thread-24] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.750 [DEBUG - New I/O worker #831] (Negotiator.java:764) Authenticated connection [id: 0xcbb95b10, /127.0.0.1:39712 => /127.22.19.253:37879] using SASL/PLAIN 00:27:35.751 [DEBUG - New I/O worker #830] (Negotiator.java:764) Authenticated connection [id: 0x1f74af24, /127.0.0.1:53082 => /127.22.19.254:44849] using SASL/PLAIN 00:27:35.751 [DEBUG - New I/O worker #889] (Negotiator.java:764) Authenticated connection [id: 0x7d10b588, /127.0.0.1:39724 => /127.22.19.253:37879] using SASL/PLAIN 00:27:35.754 [DEBUG - New I/O worker #788] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.754 [DEBUG - New I/O worker #788] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=202), Traces: [0ms] refreshing cache from master, [118ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [118ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [118ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [202ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.757 [DEBUG - New I/O worker #788] (Deferred.java:1330) callback=retry RPC@1569362645 returned Deferred@1505037756(state=PENDING, result=null, callback=(continuation of Deferred@481794286 after retry RPC@1569362645), errback=(continuation of Deferred@481794286 after retry RPC@1569362645)), so the following Deferred is getting paused: Deferred@481794286(state=PAUSED, result=Deferred@1505037756, callback=org.apache.kudu.client.AsyncKuduClient$6@24f4f6b2 -> wakeup thread pool-3-thread-58, errback=passthrough -> wakeup thread pool-3-thread-58) 00:27:35.757 [DEBUG - New I/O worker #788] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.757 [DEBUG - New I/O worker #895] (Negotiator.java:764) Authenticated connection [id: 0x9b79b3ed, /127.0.0.1:53088 => /127.22.19.254:44849] using SASL/PLAIN 00:27:35.760 [DEBUG - New I/O worker #829] (Negotiator.java:764) Authenticated connection [id: 0xd0d67c64, /127.0.0.1:37736 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.761 [DEBUG - New I/O worker #788] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.761 [DEBUG - New I/O worker #788] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.761 [DEBUG - New I/O worker #788] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.762 [DEBUG - New I/O worker #788] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.762 [DEBUG - New I/O worker #788] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.762 [DEBUG - New I/O worker #829] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.762 [DEBUG - New I/O worker #829] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=191), Traces: [0ms] refreshing cache from master, [97ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [97ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [97ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [181ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [183ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [191ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.763 [DEBUG - pool-3-thread-58] (Deferred.java:1330) callback=disconnect callback@1920146735 returned Deferred@627426(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@401423894 after disconnect callback@1920146735), errback=passthrough -> (continuation of Deferred@401423894 after disconnect callback@1920146735)), so the following Deferred is getting paused: Deferred@401423894(state=PAUSED, result=Deferred@627426, callback=, errback=) 00:27:35.763 [DEBUG - New I/O worker #829] (Deferred.java:1330) callback=retry RPC@587604943 returned Deferred@658088823(state=PENDING, result=null, callback=(continuation of Deferred@1648044532 after retry RPC@587604943), errback=(continuation of Deferred@1648044532 after retry RPC@587604943)), so the following Deferred is getting paused: Deferred@1648044532(state=PAUSED, result=Deferred@658088823, callback=org.apache.kudu.client.AsyncKuduClient$6@41195444 -> wakeup thread pool-3-thread-21, errback=passthrough -> wakeup thread pool-3-thread-21) 00:27:35.763 [DEBUG - New I/O worker #896] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.764 [DEBUG - New I/O worker #895] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.765 [DEBUG - New I/O worker #788] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.765 [DEBUG - New I/O worker #829] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:35.764998 22692 connection.cc:554] Connection torn down before Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:39718 (request call id 0) could send its response 00:27:35.769 [DEBUG - New I/O worker #829] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.769 [DEBUG - New I/O worker #829] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.770 [DEBUG - New I/O worker #829] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.770 [DEBUG - New I/O worker #829] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.770 [DEBUG - New I/O worker #829] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.771 [DEBUG - pool-3-thread-21] (Deferred.java:1330) callback=disconnect callback@972815535 returned Deferred@181158393(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1379634975 after disconnect callback@972815535), errback=passthrough -> (continuation of Deferred@1379634975 after disconnect callback@972815535)), so the following Deferred is getting paused: Deferred@1379634975(state=PAUSED, result=Deferred@181158393, callback=, errback=) 00:27:35.772 [DEBUG - New I/O worker #830] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.772 [DEBUG - New I/O worker #829] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.772 [DEBUG - New I/O worker #831] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.775 [DEBUG - New I/O worker #774] (Negotiator.java:764) Authenticated connection [id: 0xa00de42d, /127.0.0.1:37748 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.777 [DEBUG - New I/O worker #774] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.777 [DEBUG - New I/O worker #774] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=225), Traces: [0ms] refreshing cache from master, [122ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [122ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [122ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [212ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [225ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.778 [DEBUG - New I/O worker #774] (Deferred.java:1330) callback=retry RPC@863158559 returned Deferred@927810473(state=PENDING, result=null, callback=(continuation of Deferred@1917831972 after retry RPC@863158559), errback=(continuation of Deferred@1917831972 after retry RPC@863158559)), so the following Deferred is getting paused: Deferred@1917831972(state=PAUSED, result=Deferred@927810473, callback=org.apache.kudu.client.AsyncKuduClient$6@66a8d654 -> wakeup thread pool-3-thread-69, errback=passthrough -> wakeup thread pool-3-thread-69) 00:27:35.780 [DEBUG - New I/O worker #774] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.783 [DEBUG - New I/O worker #774] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.783 [DEBUG - New I/O worker #774] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.783 [DEBUG - New I/O worker #774] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.784 [DEBUG - New I/O worker #774] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.784 [DEBUG - New I/O worker #774] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.785 [DEBUG - New I/O worker #774] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:35.786674 24498 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:35.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.675389 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53094 00:27:35.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.675848 (+ 459us) server_negotiation.cc:184] Beginning negotiation 00:27:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.675875 (+ 27us) server_negotiation.cc:373] Waiting for connection header 00:27:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.675916 (+ 41us) server_negotiation.cc:381] Connection header received 00:27:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.676038 (+ 122us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.676066 (+ 28us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.676182 (+ 116us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.676360 (+ 178us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.724421 (+ 48061us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.725552 (+ 1131us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.785830 (+ 60278us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53094: BlockingRecv error: recv got EOF from 127.0.0.1:53094 (error 108) 00:27:35.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":195} 00:27:35.834 [DEBUG - pool-3-thread-71] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.834 [DEBUG - Time-limited test] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.836 [DEBUG - New I/O worker #783] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.836 [DEBUG - pool-3-thread-70] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:35.836 [DEBUG - New I/O worker #785] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.838 [DEBUG - New I/O worker #782] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.840 [DEBUG - New I/O worker #787] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.841 [DEBUG - New I/O worker #898] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.844 [DEBUG - New I/O worker #883] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.846 [DEBUG - New I/O worker #798] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.849 [DEBUG - New I/O worker #884] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.866 [DEBUG - New I/O worker #884] (Negotiator.java:764) Authenticated connection [id: 0x59c9e919, /127.0.0.1:39742 => /127.22.19.253:37879] using SASL/PLAIN 00:27:35.867 [DEBUG - New I/O worker #787] (Negotiator.java:764) Authenticated connection [id: 0x7a3e158f, /127.0.0.1:37760 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.870 [DEBUG - New I/O worker #783] (Negotiator.java:764) Authenticated connection [id: 0x63b6286c, /127.0.0.1:53100 => /127.22.19.254:44849] using SASL/PLAIN 00:27:35.871 [DEBUG - New I/O worker #899] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.873 [DEBUG - New I/O worker #787] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.873 [DEBUG - New I/O worker #787] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=125), Traces: [0ms] refreshing cache from master, [89ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [89ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [89ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [124ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.874 [DEBUG - New I/O worker #785] (Negotiator.java:764) Authenticated connection [id: 0xc47c9654, /127.0.0.1:39730 => /127.22.19.253:37879] using SASL/PLAIN 00:27:35.876 [DEBUG - New I/O worker #787] (Deferred.java:1330) callback=retry RPC@158504193 returned Deferred@223150327(state=PENDING, result=null, callback=(continuation of Deferred@1213177658 after retry RPC@158504193), errback=(continuation of Deferred@1213177658 after retry RPC@158504193)), so the following Deferred is getting paused: Deferred@1213177658(state=PAUSED, result=Deferred@223150327, callback=org.apache.kudu.client.AsyncKuduClient$6@640a3ce7 -> wakeup thread pool-3-thread-28, errback=passthrough -> wakeup thread pool-3-thread-28) 00:27:35.877 [DEBUG - New I/O worker #787] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.881 [DEBUG - New I/O worker #787] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.881 [DEBUG - New I/O worker #787] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.881 [DEBUG - New I/O worker #787] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.881 [DEBUG - New I/O worker #787] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.882 [DEBUG - New I/O worker #787] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.883 [DEBUG - New I/O worker #787] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.885 [INFO - New I/O worker #899] (Connection.java:435) [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:35.885 [DEBUG - New I/O worker #899] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:35.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:35.886759 24499 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:35.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.847501 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:39740 00:27:35.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.870689 (+ 23188us) server_negotiation.cc:184] Beginning negotiation 00:27:35.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.870717 (+ 28us) server_negotiation.cc:373] Waiting for connection header 00:27:35.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.870760 (+ 43us) server_negotiation.cc:381] Connection header received 00:27:35.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.870973 (+ 213us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:35.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.871001 (+ 28us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:35.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.871119 (+ 118us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:35.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.872705 (+ 1586us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:35.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.872964 (+ 259us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:35.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.874062 (+ 1098us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:35.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.885642 (+ 11580us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39740: BlockingRecv error: recv got EOF from 127.0.0.1:39740 (error 108) 00:27:35.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":22918} 00:27:35.887 [DEBUG - New I/O worker #883] (Negotiator.java:764) Authenticated connection [id: 0x27f3fe90, /127.0.0.1:53110 => /127.22.19.254:44849] using SASL/PLAIN 00:27:35.888 [DEBUG - New I/O worker #782] (Negotiator.java:764) Authenticated connection [id: 0xf59cf5de, /127.0.0.1:37754 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.890 [DEBUG - New I/O worker #898] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:35.890 [DEBUG - New I/O worker #798] (Negotiator.java:764) Authenticated connection [id: 0xc1d3c10b, /127.0.0.1:37762 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.891 [DEBUG - New I/O worker #782] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.891 [DEBUG - New I/O worker #782] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=142), Traces: [0ms] refreshing cache from master, [84ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [85ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [85ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [127ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [128ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [142ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.893 [DEBUG - New I/O worker #782] (Deferred.java:1330) callback=retry RPC@1392712442 returned Deferred@1463587906(state=PENDING, result=null, callback=(continuation of Deferred@306057409 after retry RPC@1392712442), errback=(continuation of Deferred@306057409 after retry RPC@1392712442)), so the following Deferred is getting paused: Deferred@306057409(state=PAUSED, result=Deferred@1463587906, callback=org.apache.kudu.client.AsyncKuduClient$6@68f940db -> wakeup thread pool-3-thread-24, errback=passthrough -> wakeup thread pool-3-thread-24) 00:27:35.895 [DEBUG - New I/O worker #798] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.895 [DEBUG - New I/O worker #798] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=149), Traces: [0ms] refreshing cache from master, [92ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [92ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [92ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [142ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [145ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [149ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.896 [DEBUG - New I/O worker #762] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.896 [DEBUG - New I/O worker #764] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.897 [DEBUG - New I/O worker #798] (Deferred.java:1330) callback=retry RPC@78850876 returned Deferred@9219954(state=PENDING, result=null, callback=(continuation of Deferred@1166942471 after retry RPC@78850876), errback=(continuation of Deferred@1166942471 after retry RPC@78850876)), so the following Deferred is getting paused: Deferred@1166942471(state=PAUSED, result=Deferred@9219954, callback=org.apache.kudu.client.AsyncKuduClient$6@69f6a960 -> wakeup thread pool-3-thread-30, errback=passthrough -> wakeup thread pool-3-thread-30) 00:27:35.898 [DEBUG - New I/O worker #782] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.898 [DEBUG - New I/O worker #798] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.906 [DEBUG - New I/O worker #763] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.909 [DEBUG - New I/O worker #782] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.910 [DEBUG - New I/O worker #782] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.910 [DEBUG - New I/O worker #782] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.910 [DEBUG - New I/O worker #782] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.910 [DEBUG - New I/O worker #782] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.911 [DEBUG - New I/O worker #798] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.911 [DEBUG - New I/O worker #798] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.912 [DEBUG - New I/O worker #798] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.912 [DEBUG - New I/O worker #798] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.912 [DEBUG - New I/O worker #798] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.913 [DEBUG - pool-3-thread-24] (Deferred.java:1330) callback=disconnect callback@1973934708 returned Deferred@207920858(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1379408987 after disconnect callback@1973934708), errback=passthrough -> (continuation of Deferred@1379408987 after disconnect callback@1973934708)), so the following Deferred is getting paused: Deferred@1379408987(state=PAUSED, result=Deferred@207920858, callback=, errback=) 00:27:35.913 [DEBUG - New I/O worker #783] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.914 [DEBUG - New I/O worker #785] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.915 [DEBUG - New I/O worker #782] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.916 [DEBUG - pool-3-thread-30] (Deferred.java:1330) callback=disconnect callback@1119654958 returned Deferred@1133392086(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@530963079 after disconnect callback@1119654958), errback=passthrough -> (continuation of Deferred@530963079 after disconnect callback@1119654958)), so the following Deferred is getting paused: Deferred@530963079(state=PAUSED, result=Deferred@1133392086, callback=, errback=) 00:27:35.916 [DEBUG - New I/O worker #884] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.917 [DEBUG - New I/O worker #762] (Negotiator.java:764) Authenticated connection [id: 0x4844441d, /127.0.0.1:37772 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.917 [DEBUG - New I/O worker #883] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.919 [DEBUG - New I/O worker #795] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.920 [DEBUG - New I/O worker #762] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.920 [DEBUG - New I/O worker #762] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=173), Traces: [0ms] refreshing cache from master, [146ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [147ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [147ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [173ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.919 [DEBUG - New I/O worker #797] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.921 [DEBUG - New I/O worker #800] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.922 [DEBUG - New I/O worker #762] (Deferred.java:1330) callback=retry RPC@1244557286 returned Deferred@1309794939(state=PENDING, result=null, callback=(continuation of Deferred@185820637 after retry RPC@1244557286), errback=(continuation of Deferred@185820637 after retry RPC@1244557286)), so the following Deferred is getting paused: Deferred@185820637(state=PAUSED, result=Deferred@1309794939, callback=org.apache.kudu.client.AsyncKuduClient$6@33a1465 -> wakeup thread pool-3-thread-78, errback=passthrough -> wakeup thread pool-3-thread-78) 00:27:35.923 [DEBUG - New I/O worker #762] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.927 [DEBUG - New I/O worker #762] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.927 [DEBUG - New I/O worker #762] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.927 [DEBUG - New I/O worker #762] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.927 [DEBUG - New I/O worker #762] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.928 [DEBUG - New I/O worker #762] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.936 [DEBUG - New I/O worker #824] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:35.937 [DEBUG - New I/O worker #825] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:35.942 [DEBUG - New I/O worker #762] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.942 [DEBUG - New I/O worker #819] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:35.953 [DEBUG - New I/O worker #800] (Negotiator.java:764) Authenticated connection [id: 0x761577b9, /127.0.0.1:39754 => /127.22.19.253:37879] using SASL/PLAIN 00:27:35.954 [DEBUG - New I/O worker #824] (Negotiator.java:764) Authenticated connection [id: 0xa1327043, /127.0.0.1:53130 => /127.22.19.254:44849] using SASL/PLAIN 00:27:35.957 [DEBUG - New I/O worker #825] (Negotiator.java:764) Authenticated connection [id: 0x4708fc3f, /127.0.0.1:39760 => /127.22.19.253:37879] using SASL/PLAIN 00:27:35.961 [INFO - New I/O worker #777] (Connection.java:435) [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:35.961 [DEBUG - New I/O worker #777] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:35.957 [DEBUG - New I/O worker #764] (Negotiator.java:764) Authenticated connection [id: 0x87122acd, /127.0.0.1:39748 => /127.22.19.253:37879] using SASL/PLAIN 00:27:35.964 [DEBUG - New I/O worker #798] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:35.967097 24622 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.895210 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53118 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.906084 (+ 10874us) server_negotiation.cc:184] Beginning negotiation 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.906107 (+ 23us) server_negotiation.cc:373] Waiting for connection header 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.906145 (+ 38us) server_negotiation.cc:381] Connection header received 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.906291 (+ 146us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.906319 (+ 28us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.906455 (+ 136us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.907856 (+ 1401us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.908183 (+ 327us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.909444 (+ 1261us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.937338 (+ 27894us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:35.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.965332 (+ 27994us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:35.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:35.965829 (+ 497us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53118: BlockingWrite error: write error: Broken pipe (error 32) 00:27:35.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":5465,"server-negotiator.queue_time_us":10575} 00:27:35.971 [DEBUG - New I/O worker #819] (Negotiator.java:764) Authenticated connection [id: 0x3f980711, /127.0.0.1:37784 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.972 [DEBUG - New I/O worker #797] (Negotiator.java:764) Authenticated connection [id: 0x653b358f, /127.0.0.1:53124 => /127.22.19.254:44849] using SASL/PLAIN 00:27:35.974 [DEBUG - New I/O worker #819] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.975 [DEBUG - New I/O worker #819] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=230), Traces: [0ms] refreshing cache from master, [188ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [189ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [189ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [211ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [219ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [229ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.975 [DEBUG - New I/O worker #819] (Deferred.java:1330) callback=retry RPC@454465193 returned Deferred@522780947(state=PENDING, result=null, callback=(continuation of Deferred@1512807570 after retry RPC@454465193), errback=(continuation of Deferred@1512807570 after retry RPC@454465193)), so the following Deferred is getting paused: Deferred@1512807570(state=PAUSED, result=Deferred@522780947, callback=org.apache.kudu.client.AsyncKuduClient$6@443777c6 -> wakeup thread pool-3-thread-10, errback=passthrough -> wakeup thread pool-3-thread-10) 00:27:35.978 [DEBUG - New I/O worker #819] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.978 [DEBUG - New I/O worker #795] (Negotiator.java:764) Authenticated connection [id: 0xcc062e0e, /127.0.0.1:37778 => /127.22.19.252:32833] using SASL/PLAIN 00:27:35.982 [DEBUG - New I/O worker #819] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.982 [DEBUG - New I/O worker #819] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.983 [DEBUG - New I/O worker #819] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.983 [DEBUG - New I/O worker #819] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.983 [DEBUG - New I/O worker #819] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.985 [DEBUG - New I/O worker #795] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:35.985 [DEBUG - New I/O worker #795] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=239), Traces: [0ms] refreshing cache from master, [170ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [170ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [170ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [215ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [229ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [239ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:35.990 [DEBUG - pool-3-thread-10] (Deferred.java:1330) callback=disconnect callback@186019131 returned Deferred@2006358796(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@981580125 after disconnect callback@186019131), errback=passthrough -> (continuation of Deferred@981580125 after disconnect callback@186019131)), so the following Deferred is getting paused: Deferred@981580125(state=PAUSED, result=Deferred@2006358796, callback=, errback=) 00:27:35.990 [DEBUG - New I/O worker #824] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:35.992 [DEBUG - New I/O worker #819] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.992 [DEBUG - New I/O worker #825] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.993 [DEBUG - New I/O worker #795] (Deferred.java:1330) callback=retry RPC@611716697 returned Deferred@541691407(state=PENDING, result=null, callback=(continuation of Deferred@1699415138 after retry RPC@611716697), errback=(continuation of Deferred@1699415138 after retry RPC@611716697)), so the following Deferred is getting paused: Deferred@1699415138(state=PAUSED, result=Deferred@541691407, callback=org.apache.kudu.client.AsyncKuduClient$6@40f48fa9 -> wakeup thread pool-3-thread-4, errback=passthrough -> wakeup thread pool-3-thread-4) 00:27:35.993 [DEBUG - New I/O worker #795] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:35.997 [DEBUG - New I/O worker #795] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:35.997 [DEBUG - New I/O worker #795] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:35.997 [DEBUG - New I/O worker #795] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:35.997 [DEBUG - New I/O worker #795] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:35.997 [DEBUG - New I/O worker #795] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:35.998 [DEBUG - New I/O worker #795] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:35.998 [DEBUG - pool-3-thread-4] (Deferred.java:1330) callback=disconnect callback@1390343312 returned Deferred@1510184336(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@961248954 after disconnect callback@1390343312), errback=passthrough -> (continuation of Deferred@961248954 after disconnect callback@1390343312)), so the following Deferred is getting paused: Deferred@961248954(state=PAUSED, result=Deferred@1510184336, callback=, errback=) 00:27:35.999 [DEBUG - New I/O worker #800] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:35.999 [DEBUG - New I/O worker #797] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.763 [INFO - New I/O worker #896] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:36.023 [DEBUG - New I/O worker #873] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.024 [DEBUG - pool-3-thread-69] (Deferred.java:1330) callback=disconnect callback@270503106 returned Deferred@1018654818(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1513530584 after disconnect callback@270503106), errback=passthrough -> (continuation of Deferred@1513530584 after disconnect callback@270503106)), so the following Deferred is getting paused: Deferred@1513530584(state=PAUSED, result=Deferred@1018654818, callback=, errback=) 00:27:36.024 [DEBUG - New I/O worker #889] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.024 [DEBUG - pool-3-thread-28] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:35.886 [INFO - New I/O worker #899] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:35.890 [INFO - New I/O worker #898] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:36.025 [DEBUG - New I/O worker #898] (Negotiator.java:764) Authenticated connection [id: 0xf3b8180b, /127.0.0.1:53108 :> /127.22.19.254:44849] using SASL/PLAIN 00:27:35.961 [INFO - New I/O worker #777] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:36.026 [DEBUG - New I/O worker #763] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:36.027 [INFO - New I/O worker #763] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:36.026 [DEBUG - pool-3-thread-78] (Deferred.java:1330) callback=disconnect callback@357274735 returned Deferred@2077936242(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1611106103 after disconnect callback@357274735), errback=passthrough -> (continuation of Deferred@1611106103 after disconnect callback@357274735)), so the following Deferred is getting paused: Deferred@1611106103(state=PAUSED, result=Deferred@2077936242, callback=, errback=) 00:27:36.027 [DEBUG - New I/O worker #764] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.028 [DEBUG - New I/O worker #920] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:36.028 [INFO - New I/O worker #920] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:36.031 [DEBUG - pool-3-thread-6] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.031 [DEBUG - New I/O worker #984] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.032 [DEBUG - New I/O worker #849] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.039 [DEBUG - New I/O worker #851] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.044 [DEBUG - New I/O worker #849] (Negotiator.java:764) Authenticated connection [id: 0x0da847c7, /127.0.0.1:37790 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.048 [DEBUG - New I/O worker #849] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.049 [DEBUG - New I/O worker #849] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=382), Traces: [0ms] refreshing cache from master, [362ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [362ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [362ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [381ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.051 [DEBUG - New I/O worker #849] (Deferred.java:1330) callback=retry RPC@1681248377 returned Deferred@1611290663(state=PENDING, result=null, callback=(continuation of Deferred@621337154 after retry RPC@1681248377), errback=(continuation of Deferred@621337154 after retry RPC@1681248377)), so the following Deferred is getting paused: Deferred@621337154(state=PAUSED, result=Deferred@1611290663, callback=org.apache.kudu.client.AsyncKuduClient$6@5e1b3d3f -> wakeup thread pool-3-thread-13, errback=passthrough -> wakeup thread pool-3-thread-13) 00:27:36.052 [DEBUG - New I/O worker #849] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.051 [DEBUG - New I/O worker #984] (Negotiator.java:764) Authenticated connection [id: 0x7089d3f3, /127.0.0.1:39766 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.057 [DEBUG - New I/O worker #851] (Negotiator.java:764) Authenticated connection [id: 0x5e7ab3fc, /127.0.0.1:53136 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.059 [DEBUG - New I/O worker #849] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:36.059 [DEBUG - New I/O worker #849] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:36.059 [DEBUG - New I/O worker #849] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:36.059 [DEBUG - New I/O worker #849] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:36.060 [DEBUG - New I/O worker #849] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.061 [DEBUG - pool-3-thread-13] (Deferred.java:1330) callback=disconnect callback@1536205824 returned Deferred@396558004(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1180207913 after disconnect callback@1536205824), errback=passthrough -> (continuation of Deferred@1180207913 after disconnect callback@1536205824)), so the following Deferred is getting paused: Deferred@1180207913(state=PAUSED, result=Deferred@396558004, callback=, errback=) 00:27:36.061 [DEBUG - New I/O worker #851] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:36.061 [DEBUG - New I/O worker #984] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.063 [DEBUG - New I/O worker #849] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.066 [DEBUG - pool-3-thread-37] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.077 [DEBUG - pool-3-thread-14] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.079 [DEBUG - pool-3-thread-38] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.083 [DEBUG - New I/O worker #1048] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.085 [DEBUG - New I/O worker #867] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.085 [DEBUG - New I/O worker #1021] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.092 [DEBUG - pool-3-thread-23] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.094 [DEBUG - pool-3-thread-32] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.095 [DEBUG - pool-3-thread-35] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.096 [DEBUG - New I/O worker #1048] (Negotiator.java:764) Authenticated connection [id: 0x18159914, /127.0.0.1:39772 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.096 [DEBUG - pool-3-thread-31] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.098 [DEBUG - pool-3-thread-33] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.098 [DEBUG - New I/O worker #842] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.100 [DEBUG - pool-3-thread-17] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.101 [DEBUG - New I/O worker #870] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.101 [DEBUG - New I/O worker #867] (Negotiator.java:764) Authenticated connection [id: 0xee71e01a, /127.0.0.1:37796 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.104 [DEBUG - New I/O worker #992] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.108 [DEBUG - New I/O worker #867] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.108 [DEBUG - New I/O worker #867] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=41), Traces: [0ms] refreshing cache from master, [12ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [13ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [13ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [32ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [41ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.115 [DEBUG - New I/O worker #867] (Deferred.java:1330) callback=retry RPC@1611240510 returned Deferred@1681263847(state=PENDING, result=null, callback=(continuation of Deferred@557096453 after retry RPC@1611240510), errback=(continuation of Deferred@557096453 after retry RPC@1611240510)), so the following Deferred is getting paused: Deferred@557096453(state=PAUSED, result=Deferred@1681263847, callback=org.apache.kudu.client.AsyncKuduClient$6@3cdba76b -> wakeup thread pool-3-thread-37, errback=passthrough -> wakeup thread pool-3-thread-37) 00:27:36.115 [DEBUG - New I/O worker #867] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.116 [DEBUG - New I/O worker #870] (Negotiator.java:764) Authenticated connection [id: 0x7438cec4, /127.0.0.1:53148 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.117 [DEBUG - New I/O worker #842] (Negotiator.java:764) Authenticated connection [id: 0x72fc5274, /127.0.0.1:37802 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.139 [DEBUG - New I/O worker #764] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.148 [DEBUG - pool-3-thread-63] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.151 [DEBUG - New I/O worker #797] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:36.155 [DEBUG - New I/O worker #825] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.160 [DEBUG - pool-3-thread-68] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.164 [DEBUG - pool-3-thread-16] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.167 [DEBUG - New I/O worker #798] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:36.172 [DEBUG - pool-3-thread-54] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.176 [DEBUG - pool-3-thread-22] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.176 [DEBUG - pool-3-thread-18] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.177 [DEBUG - pool-3-thread-5] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.180 [DEBUG - New I/O worker #831] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.180 [DEBUG - New I/O worker #782] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:36.182 [DEBUG - New I/O worker #794] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.184 [DEBUG - New I/O worker #792] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.184 [DEBUG - pool-3-thread-67] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.186 [DEBUG - New I/O worker #791] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.190 [DEBUG - New I/O worker #944] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.193 [DEBUG - pool-3-thread-55] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.194 [DEBUG - New I/O worker #936] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.194 [DEBUG - New I/O worker #756] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.195 [DEBUG - New I/O worker #752] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.195 [DEBUG - New I/O worker #860] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.196 [DEBUG - New I/O worker #999] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.196 [DEBUG - New I/O worker #933] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.197 [DEBUG - New I/O worker #755] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.198 [DEBUG - New I/O worker #748] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.200 [DEBUG - New I/O worker #943] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.203 [DEBUG - New I/O worker #751] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.204 [DEBUG - New I/O worker #850] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.205 [DEBUG - pool-3-thread-60] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.216 [DEBUG - New I/O worker #753] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.216 [DEBUG - New I/O worker #749] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.217 [DEBUG - New I/O worker #788] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:36.218 [DEBUG - pool-3-thread-3] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.223 [DEBUG - New I/O worker #754] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.224 [DEBUG - pool-3-thread-29] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.230 [DEBUG - New I/O worker #980] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.245 [DEBUG - New I/O worker #747] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.245 [DEBUG - New I/O worker #744] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.247 [DEBUG - New I/O worker #746] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.249 [DEBUG - New I/O worker #904] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.250 [DEBUG - New I/O worker #913] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.251 [DEBUG - New I/O worker #917] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.252 [DEBUG - New I/O worker #772] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.252 [DEBUG - New I/O worker #905] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.254 [DEBUG - pool-3-thread-50] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.262 [DEBUG - New I/O worker #889] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.263 [DEBUG - New I/O worker #780] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.268 [DEBUG - New I/O worker #842] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.268 [DEBUG - New I/O worker #842] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=236), Traces: [0ms] refreshing cache from master, [63ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [66ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [66ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [86ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [235ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.269 [DEBUG - New I/O worker #842] (Deferred.java:1330) callback=retry RPC@1560840137 returned Deferred@1496778928(state=PENDING, result=null, callback=(continuation of Deferred@473271794 after retry RPC@1560840137), errback=(continuation of Deferred@473271794 after retry RPC@1560840137)), so the following Deferred is getting paused: Deferred@473271794(state=PAUSED, result=Deferred@1496778928, callback=org.apache.kudu.client.AsyncKuduClient$6@344c3cd7 -> wakeup thread pool-3-thread-6, errback=passthrough -> wakeup thread pool-3-thread-6) 00:27:36.272 [DEBUG - New I/O worker #1021] (Negotiator.java:764) Authenticated connection [id: 0x8df050ec, /127.0.0.1:53142 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.272 [DEBUG - pool-3-thread-79] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.272 [DEBUG - New I/O worker #842] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.275 [DEBUG - New I/O worker #867] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:36.276 [DEBUG - New I/O worker #867] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:36.276 [DEBUG - New I/O worker #867] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:36.276 [DEBUG - New I/O worker #867] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:36.276 [DEBUG - New I/O worker #867] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.277 [DEBUG - New I/O worker #842] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:36.277 [DEBUG - New I/O worker #842] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:36.277 [DEBUG - New I/O worker #842] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:36.277 [DEBUG - pool-3-thread-37] (Deferred.java:1330) callback=disconnect callback@1785579955 returned Deferred@1446294385(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1511790231 after disconnect callback@1785579955), errback=passthrough -> (continuation of Deferred@1511790231 after disconnect callback@1785579955)), so the following Deferred is getting paused: Deferred@1511790231(state=PAUSED, result=Deferred@1446294385, callback=, errback=) 00:27:36.277 [DEBUG - New I/O worker #842] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:36.278 [DEBUG - New I/O worker #1048] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.278 [DEBUG - New I/O worker #842] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.279 [DEBUG - New I/O worker #870] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:36.280 [DEBUG - New I/O worker #842] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:36.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:36.281898 24499 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:36.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.099332 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:39778 00:27:36.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.103640 (+ 4308us) server_negotiation.cc:184] Beginning negotiation 00:27:36.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.103661 (+ 21us) server_negotiation.cc:373] Waiting for connection header 00:27:36.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.103704 (+ 43us) server_negotiation.cc:381] Connection header received 00:27:36.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.103809 (+ 105us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:36.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.103836 (+ 27us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:36.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.103959 (+ 123us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:36.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.104145 (+ 186us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:36.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.105761 (+ 1616us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:36.282 [DEBUG - New I/O worker #946] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.110071 (+ 4310us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:36.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.118097 (+ 8026us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:36.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.119402 (+ 1305us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:36.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.119612 (+ 210us) server_negotiation.cc:589] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 00:27:36.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.280802 (+161190us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39778: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:39778): Cannot send after transport endpoint shutdown (error 108) 00:27:36.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":1172} 00:27:36.283 [DEBUG - New I/O worker #1021] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:36.284 [DEBUG - New I/O worker #867] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.288 [DEBUG - New I/O worker #769] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.290 [DEBUG - pool-3-thread-53] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.291 [DEBUG - pool-3-thread-72] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.296 [DEBUG - New I/O worker #946] (Negotiator.java:764) Authenticated connection [id: 0x534fdd39, /127.0.0.1:53226 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.298 [DEBUG - New I/O worker #740] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.299 [DEBUG - New I/O worker #773] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.301 [DEBUG - New I/O worker #1155] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.301 [DEBUG - pool-3-thread-36] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.310 [DEBUG - New I/O worker #740] (Negotiator.java:764) Authenticated connection [id: 0xf2ebdd16, /127.0.0.1:37886 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.312 [DEBUG - New I/O worker #845] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.314 [DEBUG - New I/O worker #740] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.314 [DEBUG - New I/O worker #740] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=23), Traces: [0ms] refreshing cache from master, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [23ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.314 [DEBUG - New I/O worker #740] (Deferred.java:1330) callback=retry RPC@370620941 returned Deferred@304655383(state=PENDING, result=null, callback=(continuation of Deferred@1462382646 after retry RPC@370620941), errback=(continuation of Deferred@1462382646 after retry RPC@370620941)), so the following Deferred is getting paused: Deferred@1462382646(state=PAUSED, result=Deferred@304655383, callback=org.apache.kudu.client.AsyncKuduClient$6@495f3e4b -> wakeup thread pool-3-thread-53, errback=passthrough -> wakeup thread pool-3-thread-53) 00:27:36.317 [DEBUG - New I/O worker #740] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.320 [DEBUG - New I/O worker #740] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:36.321 [DEBUG - New I/O worker #740] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:36.321 [DEBUG - New I/O worker #740] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:36.321 [DEBUG - New I/O worker #740] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:36.321 [DEBUG - New I/O worker #740] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.322 [DEBUG - New I/O worker #740] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:36.325 [DEBUG - New I/O worker #849] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:36.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:36.326376 24599 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:36.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.296161 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53232 00:27:36.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.299150 (+ 2989us) server_negotiation.cc:184] Beginning negotiation 00:27:36.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.299169 (+ 19us) server_negotiation.cc:373] Waiting for connection header 00:27:36.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.299230 (+ 61us) server_negotiation.cc:381] Connection header received 00:27:36.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.299330 (+ 100us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:36.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.299383 (+ 53us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:36.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.299515 (+ 132us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:36.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.301065 (+ 1550us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:36.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.301346 (+ 281us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:36.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.303068 (+ 1722us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:36.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.314174 (+ 11106us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:36.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.324791 (+ 10617us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:36.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.325274 (+ 483us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53232: BlockingWrite error: write error: Broken pipe (error 32) 00:27:36.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":2706,"thread_start_us":2428,"threads_started":1} 00:27:36.329 [DEBUG - New I/O worker #969] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.332 [DEBUG - pool-3-thread-48] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.338 [DEBUG - New I/O worker #853] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.340 [DEBUG - New I/O worker #818] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.350 [DEBUG - New I/O worker #821] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.351 [DEBUG - New I/O worker #1063] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.353 [DEBUG - New I/O worker #1013] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.357 [DEBUG - pool-3-thread-66] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.360 [DEBUG - pool-3-thread-34] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.364 [DEBUG - pool-3-thread-19] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.366 [DEBUG - pool-3-thread-41] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.368 [DEBUG - pool-3-thread-7] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.368 [DEBUG - pool-3-thread-76] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.369 [DEBUG - pool-3-thread-9] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.370 [DEBUG - pool-3-thread-52] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.370 [DEBUG - New I/O worker #1155] (Negotiator.java:764) Authenticated connection [id: 0x64a7f420, /127.0.0.1:39856 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.371 [DEBUG - pool-3-thread-73] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.372 [DEBUG - pool-3-thread-47] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.373 [DEBUG - pool-3-thread-15] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.379 [DEBUG - New I/O worker #992] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:36.380 [INFO - New I/O worker #992] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:36.380 [DEBUG - pool-3-thread-6] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.380 [DEBUG - New I/O worker #773] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:36.383 [DEBUG - New I/O worker #1086] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.383 [DEBUG - New I/O worker #790] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.383 [DEBUG - pool-3-thread-53] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.383 [DEBUG - New I/O worker #950] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:36.384 [INFO - New I/O worker #950] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:36.385 [DEBUG - New I/O worker #1161] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.390 [DEBUG - New I/O worker #1040] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.392 [DEBUG - New I/O worker #1010] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.394 [DEBUG - New I/O worker #948] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.395 [DEBUG - New I/O worker #1008] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.397 [DEBUG - New I/O worker #918] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.398 [DEBUG - New I/O worker #775] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.399 [DEBUG - New I/O worker #809] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.403 [DEBUG - New I/O worker #1190] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.404 [DEBUG - New I/O worker #875] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.406 [DEBUG - New I/O worker #1053] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.408 [DEBUG - New I/O worker #1011] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.409 [DEBUG - New I/O worker #1005] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.409 [DEBUG - New I/O worker #914] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.409 [DEBUG - New I/O worker #1032] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.410 [INFO - New I/O worker #773] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:36.410 [DEBUG - New I/O worker #1031] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.411 [DEBUG - New I/O worker #1007] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.412 [DEBUG - New I/O worker #1061] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.421 [DEBUG - New I/O worker #853] (Negotiator.java:764) Authenticated connection [id: 0x77f06c39, /127.0.0.1:53256 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.424 [DEBUG - New I/O worker #969] (Negotiator.java:764) Authenticated connection [id: 0x6259a2d4, /127.0.0.1:53244 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.439 [DEBUG - New I/O worker #1010] (Negotiator.java:764) Authenticated connection [id: 0xcb573c85, /127.0.0.1:53238 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.440 [DEBUG - New I/O worker #1072] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.448 [DEBUG - New I/O worker #1031] (Negotiator.java:764) Authenticated connection [id: 0xf0fb7ffd, /127.0.0.1:53298 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.449 [DEBUG - New I/O worker #776] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.464 [DEBUG - New I/O worker #936] (Negotiator.java:764) Authenticated connection [id: 0x19a7cd48, /127.0.0.1:53172 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.467 [DEBUG - New I/O worker #933] (Negotiator.java:764) Authenticated connection [id: 0xcd21b3ca, /127.0.0.1:53160 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.470 [DEBUG - New I/O worker #1063] (Negotiator.java:764) Authenticated connection [id: 0xa25bbb06, /127.0.0.1:53274 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.476 [DEBUG - New I/O worker #867] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:36.484 [DEBUG - New I/O worker #755] (Negotiator.java:764) Authenticated connection [id: 0xbf6c6aea, /127.0.0.1:53166 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.486 [DEBUG - New I/O worker #949] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.489 [DEBUG - New I/O worker #1086] (Negotiator.java:764) Authenticated connection [id: 0xe3f7399a, /127.0.0.1:39850 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.490 [DEBUG - New I/O worker #845] (Negotiator.java:764) Authenticated connection [id: 0x237b7d31, /127.0.0.1:37892 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.492 [DEBUG - New I/O worker #769] (Negotiator.java:764) Authenticated connection [id: 0xdc65339c, /127.0.0.1:37880 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.494 [DEBUG - New I/O worker #784] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.500 [DEBUG - New I/O worker #875] (Negotiator.java:764) Authenticated connection [id: 0x59f2b718, /127.0.0.1:53196 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.501 [DEBUG - New I/O worker #820] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.503 [DEBUG - New I/O worker #914] (Negotiator.java:764) Authenticated connection [id: 0x2152ca68, /127.0.0.1:39838 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.503 [DEBUG - New I/O worker #1161] (Negotiator.java:764) Authenticated connection [id: 0x56de282a, /127.0.0.1:39874 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.505 [DEBUG - New I/O worker #862] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.505 [DEBUG - New I/O worker #1007] (Negotiator.java:764) Authenticated connection [id: 0xcadd0a17, /127.0.0.1:53280 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.509 [DEBUG - New I/O worker #999] (Negotiator.java:764) Authenticated connection [id: 0x20b6ff9e, /127.0.0.1:39808 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.511 [DEBUG - New I/O worker #1198] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.514 [DEBUG - New I/O worker #844] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.517 [DEBUG - New I/O worker #756] (Negotiator.java:764) Authenticated connection [id: 0x91668a62, /127.0.0.1:39796 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.518 [DEBUG - New I/O worker #845] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.518 [DEBUG - New I/O worker #845] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=421), Traces: [0ms] refreshing cache from master, [204ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [204ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [205ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [346ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [420ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.518 [DEBUG - New I/O worker #845] (Deferred.java:1330) callback=retry RPC@1624654738 returned Deferred@1693043536(state=PENDING, result=null, callback=(continuation of Deferred@569067945 after retry RPC@1624654738), errback=(continuation of Deferred@569067945 after retry RPC@1624654738)), so the following Deferred is getting paused: Deferred@569067945(state=PAUSED, result=Deferred@1693043536, callback=org.apache.kudu.client.AsyncKuduClient$6@4298fd14 -> wakeup thread pool-3-thread-31, errback=passthrough -> wakeup thread pool-3-thread-31) 00:27:36.518 [DEBUG - New I/O worker #1170] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.519 [DEBUG - New I/O worker #1072] (Negotiator.java:764) Authenticated connection [id: 0x8ac09071, /127.0.0.1:53316 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.521 [DEBUG - New I/O worker #845] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.522 [DEBUG - New I/O worker #846] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.536 [DEBUG - pool-3-thread-59] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.539 [DEBUG - New I/O worker #1013] (Negotiator.java:764) Authenticated connection [id: 0xaa7854c2, /127.0.0.1:53268 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.543 [DEBUG - New I/O worker #852] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.545 [DEBUG - New I/O worker #868] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.547 [DEBUG - New I/O worker #741] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.550 [DEBUG - New I/O worker #822] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.555 [DEBUG - New I/O worker #1065] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.557 [DEBUG - New I/O worker #1033] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.558 [DEBUG - New I/O worker #843] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:36.540102 25036 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:36.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.302374 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:39862 00:27:36.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.310965 (+ 8591us) server_negotiation.cc:184] Beginning negotiation 00:27:36.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.311006 (+ 41us) server_negotiation.cc:373] Waiting for connection header 00:27:36.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.311045 (+ 39us) server_negotiation.cc:381] Connection header received 00:27:36.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.311144 (+ 99us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:36.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.311167 (+ 23us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:36.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.311303 (+ 136us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:36.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.411620 (+100317us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39862: BlockingWrite error: write error: Broken pipe (error 32) 00:27:36.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":8320,"thread_start_us":11512,"threads_started":1} 00:27:36.559 [DEBUG - New I/O worker #1062] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.591 [DEBUG - New I/O worker #861] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.593 [DEBUG - New I/O worker #817] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.596 [DEBUG - New I/O worker #821] (Negotiator.java:764) Authenticated connection [id: 0x338034d7, /127.0.0.1:53262 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.599 [DEBUG - New I/O worker #1040] (Negotiator.java:764) Authenticated connection [id: 0xdf239ad2, /127.0.0.1:39922 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.604 [DEBUG - New I/O worker #1170] (Negotiator.java:764) Authenticated connection [id: 0xfe0327fa, /127.0.0.1:39946 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.608 [DEBUG - New I/O worker #1008] (Negotiator.java:764) Authenticated connection [id: 0x65a483b9, /127.0.0.1:39910 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.615 [DEBUG - New I/O worker #1190] (Negotiator.java:764) Authenticated connection [id: 0x4b234ea1, /127.0.0.1:39898 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.622 [DEBUG - New I/O worker #862] (Negotiator.java:764) Authenticated connection [id: 0xb73e1e3a, /127.0.0.1:39880 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.623 [DEBUG - New I/O worker #794] (Negotiator.java:764) Authenticated connection [id: 0x4c2c289e, /127.0.0.1:39784 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.625 [DEBUG - New I/O worker #1198] (Negotiator.java:764) Authenticated connection [id: 0x87625910, /127.0.0.1:39916 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.625 [DEBUG - New I/O worker #1032] (Negotiator.java:764) Authenticated connection [id: 0x675a7960, /127.0.0.1:53292 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.628 [DEBUG - New I/O worker #1061] (Negotiator.java:764) Authenticated connection [id: 0xb555cc93, /127.0.0.1:53304 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.636 [DEBUG - New I/O worker #948] (Negotiator.java:764) Authenticated connection [id: 0xfcdd2ab5, /127.0.0.1:53220 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.640 [DEBUG - New I/O worker #949] (Negotiator.java:764) Authenticated connection [id: 0xa1285cae, /127.0.0.1:39940 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.640 [DEBUG - pool-3-thread-27] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.650 [DEBUG - New I/O worker #845] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:36.651 [DEBUG - New I/O worker #845] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:36.653 [DEBUG - New I/O worker #845] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:36.653 [DEBUG - New I/O worker #845] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:36.653 [DEBUG - New I/O worker #845] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.654 [DEBUG - New I/O worker #1010] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:36.656913 24499 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.303600 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:39868 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.381530 (+ 77930us) server_negotiation.cc:184] Beginning negotiation 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.381553 (+ 23us) server_negotiation.cc:373] Waiting for connection header 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.381590 (+ 37us) server_negotiation.cc:381] Connection header received 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.381684 (+ 94us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.381707 (+ 23us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.381819 (+ 112us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.381996 (+ 177us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.419974 (+ 37978us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.493927 (+ 73953us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.655899 (+161972us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39868: BlockingRecv error: recv got EOF from 127.0.0.1:39868 (error 108) 00:27:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":31885,"server-negotiator.queue_time_us":77669,"thread_start_us":87837,"threads_started":1} 00:27:36.659 [DEBUG - New I/O worker #845] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:36.664 [DEBUG - New I/O worker #784] (Negotiator.java:764) Authenticated connection [id: 0xe27e023d, /127.0.0.1:37898 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.665 [DEBUG - New I/O worker #745] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.667 [DEBUG - New I/O worker #850] (Negotiator.java:764) Authenticated connection [id: 0x6e401e26, /127.0.0.1:37832 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.668 [DEBUG - New I/O worker #868] (Negotiator.java:764) Authenticated connection [id: 0x58ce0b3e, /127.0.0.1:37940 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.670 [DEBUG - New I/O worker #790] (Negotiator.java:764) Authenticated connection [id: 0x8477ffe9, /127.0.0.1:37856 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.670 [DEBUG - New I/O worker #743] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.671 [DEBUG - New I/O worker #748] (Negotiator.java:764) Authenticated connection [id: 0xd6b65cad, /127.0.0.1:37814 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.683 [DEBUG - New I/O worker #929] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.686 [DEBUG - New I/O worker #852] (Negotiator.java:764) Authenticated connection [id: 0x0e47dd46, /127.0.0.1:37910 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.687 [DEBUG - New I/O worker #844] (Negotiator.java:764) Authenticated connection [id: 0xfb093ad2, /127.0.0.1:37934 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.688 [DEBUG - New I/O worker #790] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.688 [DEBUG - New I/O worker #1065] (Negotiator.java:764) Authenticated connection [id: 0x8e1e3155, /127.0.0.1:39904 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.688 [DEBUG - New I/O worker #790] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=511), Traces: [0ms] refreshing cache from master, [48ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [48ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [48ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [511ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.689 [DEBUG - New I/O worker #790] (Deferred.java:1330) callback=retry RPC@895931223 returned Deferred@827932963(state=PENDING, result=null, callback=(continuation of Deferred@1952176492 after retry RPC@895931223), errback=(continuation of Deferred@1952176492 after retry RPC@895931223)), so the following Deferred is getting paused: Deferred@1952176492(state=PAUSED, result=Deferred@827932963, callback=org.apache.kudu.client.AsyncKuduClient$6@469a8606 -> wakeup thread pool-3-thread-22, errback=passthrough -> wakeup thread pool-3-thread-22) 00:27:36.690 [DEBUG - New I/O worker #1033] (Negotiator.java:764) Authenticated connection [id: 0xad1b9285, /127.0.0.1:39928 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.691 [DEBUG - New I/O worker #844] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.692 [DEBUG - New I/O worker #844] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=595), Traces: [0ms] refreshing cache from master, [250ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [250ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [250ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [412ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [514ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [595ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.692 [DEBUG - New I/O worker #844] (Deferred.java:1330) callback=retry RPC@324707905 returned Deferred@392507634(state=PENDING, result=null, callback=(continuation of Deferred@1382526074 after retry RPC@324707905), errback=(continuation of Deferred@1382526074 after retry RPC@324707905)), so the following Deferred is getting paused: Deferred@1382526074(state=PAUSED, result=Deferred@392507634, callback=org.apache.kudu.client.AsyncKuduClient$6@25d1bcff -> wakeup thread pool-3-thread-35, errback=passthrough -> wakeup thread pool-3-thread-35) 00:27:36.693 [DEBUG - New I/O worker #775] (Negotiator.java:764) Authenticated connection [id: 0x2641c492, /127.0.0.1:37868 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.694 [DEBUG - New I/O worker #809] (Negotiator.java:764) Authenticated connection [id: 0x77c6110f, /127.0.0.1:37850 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.695 [DEBUG - New I/O worker #791] (Negotiator.java:764) Authenticated connection [id: 0x35b3dc2c, /127.0.0.1:37808 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.695 [DEBUG - New I/O worker #820] (Negotiator.java:764) Authenticated connection [id: 0xefe6d356, /127.0.0.1:37916 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.696 [DEBUG - New I/O worker #790] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.697 [DEBUG - New I/O worker #844] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.698 [DEBUG - New I/O worker #739] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.699 [DEBUG - pool-3-thread-8] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.703 [DEBUG - New I/O worker #775] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.704 [DEBUG - New I/O worker #790] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:36.704 [DEBUG - New I/O worker #790] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:36.704 [DEBUG - New I/O worker #809] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.704 [DEBUG - New I/O worker #809] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=526), Traces: [0ms] refreshing cache from master, [40ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [40ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [40ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [326ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [526ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.705 [DEBUG - New I/O worker #809] (Deferred.java:1330) callback=retry RPC@1036308252 returned Deferred@972770428(state=PENDING, result=null, callback=(continuation of Deferred@2096743719 after retry RPC@1036308252), errback=(continuation of Deferred@2096743719 after retry RPC@1036308252)), so the following Deferred is getting paused: Deferred@2096743719(state=PAUSED, result=Deferred@972770428, callback=org.apache.kudu.client.AsyncKuduClient$6@5668b6f3 -> wakeup thread pool-3-thread-5, errback=passthrough -> wakeup thread pool-3-thread-5) 00:27:36.704 [DEBUG - New I/O worker #775] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=543), Traces: [0ms] refreshing cache from master, [73ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [73ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [73ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [543ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.704 [DEBUG - New I/O worker #790] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:36.706 [DEBUG - New I/O worker #790] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:36.706 [DEBUG - New I/O worker #790] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:36.708691 25003 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:36.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.226257 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53202 00:27:36.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.248355 (+ 22098us) server_negotiation.cc:184] Beginning negotiation 00:27:36.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.248400 (+ 45us) server_negotiation.cc:373] Waiting for connection header 00:27:36.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.248436 (+ 36us) server_negotiation.cc:381] Connection header received 00:27:36.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.248540 (+ 104us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:36.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.248567 (+ 27us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:36.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.248675 (+ 108us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:36.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.249761 (+ 1086us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:36.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.707591 (+457830us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53202: BlockingRecv error: recv got EOF from 127.0.0.1:53202 (error 108) 00:27:36.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":21770,"thread_start_us":5061,"threads_started":1} 00:27:36.710 [DEBUG - New I/O worker #790] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:36.713 [DEBUG - New I/O worker #767] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.723 [DEBUG - New I/O worker #847] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.724 [DEBUG - New I/O worker #868] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.724 [DEBUG - New I/O worker #868] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=646), Traces: [0ms] refreshing cache from master, [275ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [275ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [275ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [550ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [647ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.725 [DEBUG - New I/O worker #852] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.726 [DEBUG - New I/O worker #852] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=627), Traces: [0ms] refreshing cache from master, [228ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [228ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [228ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [326ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [627ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.726 [DEBUG - New I/O worker #852] (Deferred.java:1330) callback=retry RPC@399610851 returned Deferred@334370241(state=PENDING, result=null, callback=(continuation of Deferred@1458346456 after retry RPC@399610851), errback=(continuation of Deferred@1458346456 after retry RPC@399610851)), so the following Deferred is getting paused: Deferred@1458346456(state=PAUSED, result=Deferred@334370241, callback=org.apache.kudu.client.AsyncKuduClient$6@7c258642 -> wakeup thread pool-3-thread-33, errback=passthrough -> wakeup thread pool-3-thread-33) 00:27:36.727 [DEBUG - New I/O worker #791] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.727 [DEBUG - New I/O worker #791] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=550), Traces: [0ms] refreshing cache from master, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [472ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [551ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.728 [DEBUG - New I/O worker #791] (Deferred.java:1330) callback=retry RPC@465445951 returned Deferred@528589712(state=PENDING, result=null, callback=(continuation of Deferred@1518546436 after retry RPC@465445951), errback=(continuation of Deferred@1518546436 after retry RPC@465445951)), so the following Deferred is getting paused: Deferred@1518546436(state=PAUSED, result=Deferred@528589712, callback=org.apache.kudu.client.AsyncKuduClient$6@1e6d37c -> wakeup thread pool-3-thread-18, errback=passthrough -> wakeup thread pool-3-thread-18) 00:27:36.728 [DEBUG - New I/O worker #860] (Negotiator.java:764) Authenticated connection [id: 0x1788dbe3, /127.0.0.1:53178 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.729 [DEBUG - New I/O worker #850] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.729 [DEBUG - New I/O worker #850] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=894), Traces: [0ms] refreshing cache from master, [349ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [354ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [355ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [714ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [894ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.730 [DEBUG - New I/O worker #868] (Deferred.java:1330) callback=retry RPC@170033341 returned Deferred@236786977(state=PENDING, result=null, callback=(continuation of Deferred@1260358278 after retry RPC@170033341), errback=(continuation of Deferred@1260358278 after retry RPC@170033341)), so the following Deferred is getting paused: Deferred@1260358278(state=PAUSED, result=Deferred@236786977, callback=org.apache.kudu.client.AsyncKuduClient$6@64277242 -> wakeup thread pool-3-thread-14, errback=passthrough -> wakeup thread pool-3-thread-14) 00:27:36.730 [DEBUG - New I/O worker #850] (Deferred.java:1330) callback=retry RPC@104557776 returned Deferred@33876175(state=PENDING, result=null, callback=(continuation of Deferred@1191601899 after retry RPC@104557776), errback=(continuation of Deferred@1191601899 after retry RPC@104557776)), so the following Deferred is getting paused: Deferred@1191601899(state=PAUSED, result=Deferred@33876175, callback=org.apache.kudu.client.AsyncKuduClient$6@2b8bf4cf -> wakeup thread Time-limited test, errback=passthrough -> wakeup thread Time-limited test) 00:27:36.738 [INFO - New I/O worker #1011] (Connection.java:435) [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:36.738 [DEBUG - New I/O worker #1011] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:36.738 [INFO - New I/O worker #1011] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:36.746 [DEBUG - New I/O worker #769] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.746 [DEBUG - New I/O worker #775] (Deferred.java:1330) callback=retry RPC@1373732892 returned Deferred@1440620280(state=PENDING, result=null, callback=(continuation of Deferred@282888743 after retry RPC@1373732892), errback=(continuation of Deferred@282888743 after retry RPC@1373732892)), so the following Deferred is getting paused: Deferred@282888743(state=PAUSED, result=Deferred@1440620280, callback=org.apache.kudu.client.AsyncKuduClient$6@7c3a2781 -> wakeup thread pool-3-thread-68, errback=passthrough -> wakeup thread pool-3-thread-68) 00:27:36.756 [DEBUG - New I/O worker #745] (Negotiator.java:764) Authenticated connection [id: 0xb04fff07, /127.0.0.1:39952 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.756 [DEBUG - New I/O worker #809] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.757 [DEBUG - pool-3-thread-75] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.758 [DEBUG - New I/O worker #739] (Negotiator.java:764) Authenticated connection [id: 0x3a423cc1, /127.0.0.1:37976 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.759 [DEBUG - New I/O worker #817] (Negotiator.java:764) Authenticated connection [id: 0xa9a1990a, /127.0.0.1:37904 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.759 [DEBUG - New I/O worker #861] (Negotiator.java:764) Authenticated connection [id: 0x8947c2ac, /127.0.0.1:39892 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.760 [DEBUG - New I/O worker #930] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.762 [DEBUG - New I/O worker #822] (Negotiator.java:764) Authenticated connection [id: 0xf33323b0, /127.0.0.1:37946 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.763 [DEBUG - New I/O worker #743] (Negotiator.java:764) Authenticated connection [id: 0x4d5be5cb, /127.0.0.1:53322 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.769 [DEBUG - New I/O worker #741] (Negotiator.java:764) Authenticated connection [id: 0xe4b2b563, /127.0.0.1:37964 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.771 [DEBUG - New I/O worker #847] (Negotiator.java:764) Authenticated connection [id: 0x8cbc7ec7, /127.0.0.1:37958 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.771 [DEBUG - New I/O worker #809] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:36.771 [DEBUG - New I/O worker #809] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:36.772 [DEBUG - New I/O worker #809] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:36.772 [DEBUG - New I/O worker #809] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:36.772 [DEBUG - New I/O worker #792] (Negotiator.java:764) Authenticated connection [id: 0x6519b46a, /127.0.0.1:53154 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.775 [DEBUG - New I/O worker #847] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.775 [DEBUG - New I/O worker #741] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.776 [DEBUG - New I/O worker #741] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=611), Traces: [0ms] refreshing cache from master, [236ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [237ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [237ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [479ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [611ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.772 [DEBUG - New I/O worker #817] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.848 [DEBUG - New I/O worker #1204] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.849 [DEBUG - New I/O worker #741] (Deferred.java:1330) callback=retry RPC@1923919763 returned Deferred@1989358878(state=PENDING, result=null, callback=(continuation of Deferred@865183144 after retry RPC@1923919763), errback=(continuation of Deferred@865183144 after retry RPC@1923919763)), so the following Deferred is getting paused: Deferred@865183144(state=PAUSED, result=Deferred@1989358878, callback=org.apache.kudu.client.AsyncKuduClient$6@152f059a -> wakeup thread pool-3-thread-16, errback=passthrough -> wakeup thread pool-3-thread-16) 00:27:36.847 [DEBUG - New I/O worker #1049] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.851 [DEBUG - New I/O worker #866] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.846 [DEBUG - New I/O worker #739] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.845 [DEBUG - New I/O worker #775] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.845 [DEBUG - New I/O worker #748] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.878 [DEBUG - New I/O worker #748] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1043), Traces: [0ms] refreshing cache from master, [350ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [351ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [351ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [635ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [913ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.878 [DEBUG - New I/O worker #748] (Deferred.java:1330) callback=retry RPC@1302362757 returned Deferred@1235220457(state=PENDING, result=null, callback=(continuation of Deferred@211644606 after retry RPC@1302362757), errback=(continuation of Deferred@211644606 after retry RPC@1302362757)), so the following Deferred is getting paused: Deferred@211644606(state=PAUSED, result=Deferred@1235220457, callback=org.apache.kudu.client.AsyncKuduClient$6@73e1a6e0 -> wakeup thread pool-3-thread-71, errback=passthrough -> wakeup thread pool-3-thread-71) 00:27:36.845 [DEBUG - New I/O worker #820] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.879 [DEBUG - New I/O worker #820] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=784), Traces: [0ms] refreshing cache from master, [233ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [234ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [234ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [514ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [658ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [668ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:36.845 [DEBUG - New I/O worker #1017] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.845 [DEBUG - New I/O worker #1019] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.880 [DEBUG - New I/O worker #744] (Negotiator.java:764) Authenticated connection [id: 0x102d6253, /127.0.0.1:37844 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.844 [DEBUG - New I/O worker #944] (Negotiator.java:764) Authenticated connection [id: 0x4dbeee60, /127.0.0.1:39790 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.843 [DEBUG - pool-3-thread-44] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.842 [DEBUG - pool-3-thread-51] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.842 [DEBUG - New I/O worker #809] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299930, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299930, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299930, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299930, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.841 [DEBUG - pool-3-thread-49] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.884 [DEBUG - New I/O worker #744] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.884 [DEBUG - New I/O worker #744] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=691), Traces: [0ms] refreshing cache from master, [23ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [23ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [24ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [690ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.885 [DEBUG - New I/O worker #817] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=784), Traces: [0ms] refreshing cache from master, [225ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [225ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [225ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [547ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [672ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.841 [DEBUG - New I/O worker #930] (Negotiator.java:764) Authenticated connection [id: 0xaaa0691b, /127.0.0.1:39958 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.840 [DEBUG - pool-3-thread-1] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.840 [DEBUG - New I/O worker #837] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.887 [DEBUG - New I/O worker #744] (Deferred.java:1330) callback=retry RPC@1739059339 returned Deferred@1670932647(state=PENDING, result=null, callback=(continuation of Deferred@647690928 after retry RPC@1739059339), errback=(continuation of Deferred@647690928 after retry RPC@1739059339)), so the following Deferred is getting paused: Deferred@647690928(state=PAUSED, result=Deferred@1670932647, callback=org.apache.kudu.client.AsyncKuduClient$6@4f8d2b0f -> wakeup thread pool-3-thread-55, errback=passthrough -> wakeup thread pool-3-thread-55) 00:27:36.839 [DEBUG - New I/O worker #844] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:36.889 [DEBUG - New I/O worker #844] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:36.889 [DEBUG - New I/O worker #844] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:36.889 [DEBUG - New I/O worker #844] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:36.890 [DEBUG - New I/O worker #844] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.890 [DEBUG - pool-3-thread-35] (Deferred.java:1330) callback=disconnect callback@1026492712 returned Deferred@971954632(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@797603069 after disconnect callback@1026492712), errback=passthrough -> (continuation of Deferred@797603069 after disconnect callback@1026492712)), so the following Deferred is getting paused: Deferred@797603069(state=PAUSED, result=Deferred@971954632, callback=, errback=) 00:27:36.891 [DEBUG - New I/O worker #1007] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:36.891 [DEBUG - New I/O worker #1008] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.892 [DEBUG - New I/O worker #844] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.838 [DEBUG - New I/O worker #784] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.838 [DEBUG - New I/O worker #868] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.838 [DEBUG - New I/O worker #791] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.838 [DEBUG - New I/O worker #852] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.894 [DEBUG - New I/O worker #744] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.838 [DEBUG - New I/O worker #850] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.838 [DEBUG - New I/O worker #769] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=564), Traces: [0ms] refreshing cache from master, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [5ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [7ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [26ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [359ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [447ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.838 [DEBUG - New I/O worker #1062] (Negotiator.java:764) Authenticated connection [id: 0x162f1028, /127.0.0.1:39934 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.896 [DEBUG - New I/O worker #769] (Deferred.java:1330) callback=retry RPC@1627416888 returned Deferred@1698689472(state=PENDING, result=null, callback=(continuation of Deferred@540893955 after retry RPC@1627416888), errback=(continuation of Deferred@540893955 after retry RPC@1627416888)), so the following Deferred is getting paused: Deferred@540893955(state=PAUSED, result=Deferred@1698689472, callback=org.apache.kudu.client.AsyncKuduClient$6@737c7fae -> wakeup thread pool-3-thread-79, errback=passthrough -> wakeup thread pool-3-thread-79) 00:27:36.834 [DEBUG - New I/O worker #1023] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.833 [DEBUG - New I/O worker #1046] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.898 [DEBUG - New I/O worker #744] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:36.833 [DEBUG - New I/O worker #753] (Negotiator.java:764) Authenticated connection [id: 0x4015d53e, /127.0.0.1:53184 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.831 [DEBUG - New I/O worker #943] (Negotiator.java:764) Authenticated connection [id: 0xb4e0b9fe, /127.0.0.1:39802 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.830 [DEBUG - New I/O worker #865] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.830 [DEBUG - New I/O worker #1025] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.829 [DEBUG - New I/O worker #1026] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.901 [DEBUG - New I/O worker #784] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=610), Traces: [0ms] refreshing cache from master, [12ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [12ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [12ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [135ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [356ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [454ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.828 [DEBUG - New I/O worker #1044] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.828 [DEBUG - New I/O worker #1024] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.823 [DEBUG - New I/O worker #1029] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.823 [DEBUG - New I/O worker #858] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.905 [DEBUG - New I/O worker #767] (Negotiator.java:764) Authenticated connection [id: 0xff7bdc98, /127.0.0.1:37982 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.823 [DEBUG - New I/O worker #857] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.908 [DEBUG - New I/O worker #784] (Deferred.java:1330) callback=retry RPC@748836578 returned Deferred@681428910(state=PENDING, result=null, callback=(continuation of Deferred@1839160537 after retry RPC@748836578), errback=(continuation of Deferred@1839160537 after retry RPC@748836578)), so the following Deferred is getting paused: Deferred@1839160537(state=PAUSED, result=Deferred@681428910, callback=org.apache.kudu.client.AsyncKuduClient$6@62931ee8 -> wakeup thread pool-3-thread-72, errback=passthrough -> wakeup thread pool-3-thread-72) 00:27:36.908 [DEBUG - New I/O worker #1204] (Negotiator.java:764) Authenticated connection [id: 0xc65f5be1, /127.0.0.1:40036 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.913 [DEBUG - New I/O worker #769] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.822 [DEBUG - New I/O worker #846] (Negotiator.java:764) Authenticated connection [id: 0xf402c3c0, /127.0.0.1:37928 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.820 [DEBUG - New I/O worker #1053] (Negotiator.java:764) Authenticated connection [id: 0x49b97e4e, /127.0.0.1:53286 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.819 [DEBUG - New I/O worker #789] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.917 [DEBUG - New I/O worker #866] (Negotiator.java:764) Authenticated connection [id: 0xa7f902e5, /127.0.0.1:38060 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.818 [DEBUG - New I/O worker #1079] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.918 [DEBUG - New I/O worker #769] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:36.918 [DEBUG - New I/O worker #769] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:36.918 [DEBUG - New I/O worker #769] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:36.918 [DEBUG - New I/O worker #769] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:36.919 [DEBUG - New I/O worker #769] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.919 [DEBUG - New I/O worker #980] (Negotiator.java:764) Authenticated connection [id: 0x32621153, /127.0.0.1:39826 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.920 [DEBUG - pool-3-thread-79] (Deferred.java:1330) callback=disconnect callback@1031371784 returned Deferred@825217047(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1946354526 after disconnect callback@1031371784), errback=passthrough -> (continuation of Deferred@1946354526 after disconnect callback@1031371784)), so the following Deferred is getting paused: Deferred@1946354526(state=PAUSED, result=Deferred@825217047, callback=, errback=) 00:27:36.920 [DEBUG - New I/O worker #1155] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.920 [DEBUG - New I/O worker #946] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:36.921 [DEBUG - New I/O worker #769] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.817 [DEBUG - New I/O worker #918] (Negotiator.java:764) Authenticated connection [id: 0xee9289ac, /127.0.0.1:39844 => /127.22.19.253:37879] using SASL/PLAIN 00:27:36.816 [DEBUG - New I/O worker #1042] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.815 [DEBUG - New I/O worker #1027] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.815 [DEBUG - New I/O worker #838] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.813 [DEBUG - New I/O worker #818] (Negotiator.java:764) Authenticated connection [id: 0x0e4bb690, /127.0.0.1:53250 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.812 [DEBUG - New I/O worker #882] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.812 [DEBUG - New I/O worker #869] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.928 [DEBUG - pool-3-thread-5] (Deferred.java:1330) callback=disconnect callback@299436361 returned Deferred@185264671(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@571632206 after disconnect callback@299436361), errback=passthrough -> (continuation of Deferred@571632206 after disconnect callback@299436361)), so the following Deferred is getting paused: Deferred@571632206(state=PAUSED, result=Deferred@185264671, callback=, errback=) 00:27:36.928 [DEBUG - New I/O worker #875] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:36.810 [DEBUG - New I/O worker #931] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.928 [DEBUG - New I/O worker #809] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:36.810 [DEBUG - New I/O worker #932] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.809 [DEBUG - New I/O worker #799] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.809 [DEBUG - New I/O worker #765] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.933 [DEBUG - New I/O worker #980] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.935 [DEBUG - New I/O worker #1020] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.807 [DEBUG - New I/O worker #751] (Negotiator.java:764) Authenticated connection [id: 0x6793626d, /127.0.0.1:37826 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.806 [DEBUG - New I/O worker #919] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.805 [DEBUG - New I/O worker #947] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.805 [DEBUG - New I/O worker #786] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.982 [DEBUG - New I/O worker #850] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:36.982 [DEBUG - New I/O worker #850] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:36.982 [DEBUG - New I/O worker #850] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:36.983 [DEBUG - New I/O worker #850] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:36.983 [DEBUG - New I/O worker #850] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.984 [DEBUG - New I/O worker #868] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:36.984 [DEBUG - New I/O worker #868] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:36.984 [DEBUG - New I/O worker #868] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:36.985 [DEBUG - New I/O worker #868] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:36.985 [DEBUG - New I/O worker #868] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.986 [DEBUG - pool-3-thread-14] (Deferred.java:1330) callback=disconnect callback@742355722 returned Deferred@125144174(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@378350946 after disconnect callback@742355722), errback=passthrough -> (continuation of Deferred@378350946 after disconnect callback@742355722)), so the following Deferred is getting paused: Deferred@378350946(state=PAUSED, result=Deferred@125144174, callback=, errback=) 00:27:36.986 [DEBUG - Time-limited test] (Deferred.java:1330) callback=disconnect callback@1955528702 returned Deferred@765111187(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1861469328 after disconnect callback@1955528702), errback=passthrough -> (continuation of Deferred@1861469328 after disconnect callback@1955528702)), so the following Deferred is getting paused: Deferred@1861469328(state=PAUSED, result=Deferred@765111187, callback=, errback=) 00:27:36.987 [DEBUG - New I/O worker #1053] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:36.987 [DEBUG - New I/O worker #1198] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.987 [DEBUG - New I/O worker #860] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:36.988 [DEBUG - New I/O worker #999] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.988 [DEBUG - New I/O worker #784] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.990 [DEBUG - New I/O worker #868] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.991 [DEBUG - New I/O worker #850] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.801 [DEBUG - New I/O worker #749] (Negotiator.java:764) Authenticated connection [id: 0x1355b414, /127.0.0.1:37838 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.800 [DEBUG - New I/O worker #1012] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.799 [DEBUG - New I/O worker #854] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.799 [DEBUG - New I/O worker #856] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.798 [DEBUG - New I/O worker #1005] (Negotiator.java:764) Authenticated connection [id: 0x10896f8f, /127.0.0.1:39886 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.020 [DEBUG - New I/O worker #769] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:36.798 [DEBUG - New I/O worker #859] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.797 [DEBUG - New I/O worker #812] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.795 [DEBUG - New I/O worker #822] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.794 [DEBUG - New I/O worker #752] (Negotiator.java:764) Authenticated connection [id: 0xb3d972c0, /127.0.0.1:37820 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.790 [DEBUG - pool-3-thread-57] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:36.788 [DEBUG - New I/O worker #816] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.787 [DEBUG - New I/O worker #1015] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.786 [DEBUG - New I/O worker #863] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.786 [DEBUG - New I/O worker #897] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:36.786 [DEBUG - New I/O worker #966] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:36.783581 24989 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:36.781 [DEBUG - pool-3-thread-31] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.776 [DEBUG - New I/O worker #843] (Negotiator.java:764) Authenticated connection [id: 0xb342da40, /127.0.0.1:37922 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.775 [DEBUG - New I/O worker #847] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=442), Traces: [0ms] refreshing cache from master, [39ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [40ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [40ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [297ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [442ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.057 [DEBUG - New I/O worker #917] (Negotiator.java:764) Authenticated connection [id: 0xde518cfe, /127.0.0.1:53214 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.060 [DEBUG - New I/O worker #1023] (Negotiator.java:764) Authenticated connection [id: 0xf47bd839, /127.0.0.1:53396 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.056 [DEBUG - New I/O worker #932] (Negotiator.java:764) Authenticated connection [id: 0x2ff936c3, /127.0.0.1:39990 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.055 [DEBUG - New I/O worker #754] (Negotiator.java:764) Authenticated connection [id: 0x3b029f6f, /127.0.0.1:39814 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.084 [DEBUG - New I/O worker #850] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.051 [DEBUG - New I/O worker #752] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.050 [DEBUG - New I/O worker #844] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.226069 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:39832 00:27:37.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.252153 (+ 26084us) server_negotiation.cc:184] Beginning negotiation 00:27:37.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.252179 (+ 26us) server_negotiation.cc:373] Waiting for connection header 00:27:37.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.252240 (+ 61us) server_negotiation.cc:381] Connection header received 00:27:37.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.252349 (+ 109us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:37.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.252374 (+ 25us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:37.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.252502 (+ 128us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:37.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.253024 (+ 522us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:37.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:36.782402 (+529378us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39832: BlockingRecv error: recv got EOF from 127.0.0.1:39832 (error 108) 00:27:37.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":25792,"thread_start_us":3186,"threads_started":1} 00:27:37.043 [DEBUG - New I/O worker #822] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=682), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [247ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [270ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [406ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.041 [DEBUG - New I/O worker #866] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.135 [DEBUG - New I/O worker #1028] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.039 [DEBUG - New I/O worker #751] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.039 [DEBUG - New I/O worker #747] (Negotiator.java:764) Authenticated connection [id: 0x2e462fbb, /127.0.0.1:39820 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.030 [DEBUG - New I/O worker #746] (Negotiator.java:764) Authenticated connection [id: 0x954ddff0, /127.0.0.1:53190 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.028 [DEBUG - New I/O worker #784] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.156 [DEBUG - New I/O worker #784] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.157 [DEBUG - New I/O worker #784] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.157 [DEBUG - New I/O worker #784] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.158 [DEBUG - New I/O worker #1030] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.021 [DEBUG - New I/O worker #780] (Negotiator.java:764) Authenticated connection [id: 0x115032d0, /127.0.0.1:37874 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.019 [DEBUG - New I/O worker #1049] (Negotiator.java:764) Authenticated connection [id: 0x151aa0f2, /127.0.0.1:53406 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.163 [DEBUG - New I/O worker #1020] (Negotiator.java:764) Authenticated connection [id: 0xba91d72f, /127.0.0.1:53412 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.996 [DEBUG - New I/O worker #846] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:36.984 [DEBUG - New I/O worker #775] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.186 [DEBUG - New I/O worker #775] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.187 [DEBUG - New I/O worker #775] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.187 [DEBUG - New I/O worker #775] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.187 [DEBUG - New I/O worker #775] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.978 [DEBUG - New I/O worker #852] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.196 [DEBUG - New I/O worker #852] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.214 [DEBUG - New I/O worker #852] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.214 [DEBUG - New I/O worker #852] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.214 [DEBUG - New I/O worker #852] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.215 [DEBUG - New I/O worker #919] (Negotiator.java:764) Authenticated connection [id: 0x4ed74105, /127.0.0.1:53354 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.977 [DEBUG - New I/O worker #791] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.215 [DEBUG - pool-3-thread-33] (Deferred.java:1330) callback=disconnect callback@1375942562 returned Deferred@771801395(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@741657996 after disconnect callback@1375942562), errback=passthrough -> (continuation of Deferred@741657996 after disconnect callback@1375942562)), so the following Deferred is getting paused: Deferred@741657996(state=PAUSED, result=Deferred@771801395, callback=, errback=) 00:27:37.216 [DEBUG - New I/O worker #853] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.216 [DEBUG - New I/O worker #1005] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.977 [DEBUG - New I/O worker #748] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.959 [DEBUG - New I/O worker #767] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.217 [DEBUG - New I/O worker #767] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=962), Traces: [0ms] refreshing cache from master, [425ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [425ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [426ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [661ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [696ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.217 [DEBUG - New I/O worker #767] (Deferred.java:1330) callback=retry RPC@1661836162 returned Deferred@1731330875(state=PENDING, result=null, callback=(continuation of Deferred@573608377 after retry RPC@1661836162), errback=(continuation of Deferred@573608377 after retry RPC@1661836162)), so the following Deferred is getting paused: Deferred@573608377(state=PAUSED, result=Deferred@1731330875, callback=org.apache.kudu.client.AsyncKuduClient$6@1c863d84 -> wakeup thread pool-3-thread-50, errback=passthrough -> wakeup thread pool-3-thread-50) 00:27:36.919 [DEBUG - New I/O worker #913] (Negotiator.java:764) Authenticated connection [id: 0x73be4ca9, /127.0.0.1:53208 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.918 [DEBUG - New I/O worker #772] (Negotiator.java:764) Authenticated connection [id: 0x49f42748, /127.0.0.1:37862 => /127.22.19.252:32833] using SASL/PLAIN 00:27:36.904 [DEBUG - New I/O worker #817] (Deferred.java:1330) callback=retry RPC@345705687 returned Deferred@279222940(state=PENDING, result=null, callback=(continuation of Deferred@1436944108 after retry RPC@345705687), errback=(continuation of Deferred@1436944108 after retry RPC@345705687)), so the following Deferred is getting paused: Deferred@1436944108(state=PAUSED, result=Deferred@279222940, callback=org.apache.kudu.client.AsyncKuduClient$6@28885e25 -> wakeup thread pool-3-thread-17, errback=passthrough -> wakeup thread pool-3-thread-17) 00:27:37.219 [DEBUG - New I/O worker #817] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.899 [DEBUG - New I/O worker #744] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.220 [DEBUG - New I/O worker #744] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.221 [DEBUG - New I/O worker #744] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.221 [DEBUG - New I/O worker #744] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:36.880 [DEBUG - New I/O worker #820] (Deferred.java:1330) callback=retry RPC@341018810 returned Deferred@275521256(state=PENDING, result=null, callback=(continuation of Deferred@1433305729 after retry RPC@341018810), errback=(continuation of Deferred@1433305729 after retry RPC@341018810)), so the following Deferred is getting paused: Deferred@1433305729(state=PAUSED, result=Deferred@275521256, callback=org.apache.kudu.client.AsyncKuduClient$6@6cef1501 -> wakeup thread pool-3-thread-32, errback=passthrough -> wakeup thread pool-3-thread-32) 00:27:37.222 [DEBUG - pool-3-thread-55] (Deferred.java:1330) callback=disconnect callback@733061222 returned Deferred@117907705(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@397970159 after disconnect callback@733061222), errback=passthrough -> (continuation of Deferred@397970159 after disconnect callback@733061222)), so the following Deferred is getting paused: Deferred@397970159(state=PAUSED, result=Deferred@117907705, callback=, errback=) 00:27:37.222 [DEBUG - New I/O worker #746] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.222 [DEBUG - New I/O worker #747] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:36.878 [DEBUG - New I/O worker #839] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.877 [DEBUG - New I/O worker #739] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=658), Traces: [0ms] refreshing cache from master, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [422ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [540ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [547ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [551ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:36.877 [DEBUG - New I/O worker #823] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:36.876 [DEBUG - New I/O worker #741] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:36.875 [DEBUG - New I/O worker #1022] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:36.871 [DEBUG - New I/O worker #929] (Negotiator.java:764) Authenticated connection [id: 0xd70d13e8, /127.0.0.1:53328 => /127.22.19.254:44849] using SASL/PLAIN 00:27:36.867 [INFO - New I/O worker #905] (Connection.java:435) [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:37.230 [DEBUG - New I/O worker #905] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:37.230 [INFO - New I/O worker #905] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:36.864 [INFO - New I/O worker #904] (Connection.java:435) [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:37.231 [DEBUG - New I/O worker #904] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:37.231 [INFO - New I/O worker #904] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:36.862 [DEBUG - pool-3-thread-22] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:36.848 [DEBUG - New I/O worker #776] (Negotiator.java:764) Authenticated connection [id: 0xb49db471, /127.0.0.1:53310 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.237 [DEBUG - New I/O worker #739] (Deferred.java:1330) callback=retry RPC@1242253530 returned Deferred@1312085315(state=PENDING, result=null, callback=(continuation of Deferred@188105441 after retry RPC@1242253530), errback=(continuation of Deferred@188105441 after retry RPC@1242253530)), so the following Deferred is getting paused: Deferred@188105441(state=PAUSED, result=Deferred@1312085315, callback=org.apache.kudu.client.AsyncKuduClient$6@2a2a4892 -> wakeup thread pool-3-thread-3, errback=passthrough -> wakeup thread pool-3-thread-3) 00:27:37.243 [DEBUG - New I/O worker #748] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.244 [DEBUG - New I/O worker #748] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.244 [DEBUG - New I/O worker #748] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.244 [DEBUG - New I/O worker #748] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.244 [DEBUG - New I/O worker #748] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.246 [DEBUG - New I/O worker #852] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.227 [DEBUG - New I/O worker #744] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.222 [DEBUG - New I/O worker #820] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.215 [DEBUG - New I/O worker #791] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.248 [DEBUG - New I/O worker #791] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.248 [DEBUG - New I/O worker #791] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.248 [DEBUG - New I/O worker #791] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.213 [DEBUG - New I/O worker #1222] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.213 [DEBUG - New I/O worker #841] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.249 [DEBUG - pool-3-thread-18] (Deferred.java:1330) callback=disconnect callback@1133466130 returned Deferred@1070963789(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2001538757 after disconnect callback@1133466130), errback=passthrough -> (continuation of Deferred@2001538757 after disconnect callback@1133466130)), so the following Deferred is getting paused: Deferred@2001538757(state=PAUSED, result=Deferred@1070963789, callback=, errback=) 00:27:37.212 [DEBUG - New I/O worker #1057] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.212 [DEBUG - New I/O worker #928] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.212 [DEBUG - New I/O worker #1036] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.212 [DEBUG - New I/O worker #848] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.212 [DEBUG - New I/O worker #742] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.251 [DEBUG - New I/O worker #741] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.251 [DEBUG - New I/O worker #741] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.252 [DEBUG - New I/O worker #741] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.252 [DEBUG - New I/O worker #741] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.252 [DEBUG - New I/O worker #741] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.253 [DEBUG - pool-3-thread-16] (Deferred.java:1330) callback=disconnect callback@425116240 returned Deferred@1766389934(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@253321164 after disconnect callback@425116240), errback=passthrough -> (continuation of Deferred@253321164 after disconnect callback@425116240)), so the following Deferred is getting paused: Deferred@253321164(state=PAUSED, result=Deferred@1766389934, callback=, errback=) 00:27:37.253 [DEBUG - New I/O worker #949] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.254 [DEBUG - New I/O worker #776] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.255 [DEBUG - New I/O worker #741] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.257 [DEBUG - New I/O worker #794] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.258 [DEBUG - New I/O worker #767] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.258 [DEBUG - New I/O worker #792] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.262 [DEBUG - New I/O worker #767] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.263 [DEBUG - New I/O worker #767] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.263 [DEBUG - New I/O worker #767] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.263 [DEBUG - New I/O worker #767] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.263 [DEBUG - New I/O worker #767] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.264 [DEBUG - New I/O worker #767] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.270 [DEBUG - New I/O worker #820] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.271 [DEBUG - New I/O worker #820] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.271 [DEBUG - New I/O worker #820] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.271 [DEBUG - New I/O worker #820] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.272 [DEBUG - New I/O worker #820] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.272 [DEBUG - pool-3-thread-32] (Deferred.java:1330) callback=disconnect callback@346208513 returned Deferred@761632074(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1237550933 after disconnect callback@346208513), errback=passthrough -> (continuation of Deferred@1237550933 after disconnect callback@346208513)), so the following Deferred is getting paused: Deferred@1237550933(state=PAUSED, result=Deferred@761632074, callback=, errback=) 00:27:37.273 [DEBUG - New I/O worker #821] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.273 [DEBUG - New I/O worker #861] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.273 [DEBUG - New I/O worker #791] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.274 [DEBUG - New I/O worker #820] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.279 [DEBUG - pool-3-thread-71] (Deferred.java:1330) callback=disconnect callback@953050826 returned Deferred@615296545(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@555673659 after disconnect callback@953050826), errback=passthrough -> (continuation of Deferred@555673659 after disconnect callback@953050826)), so the following Deferred is getting paused: Deferred@555673659(state=PAUSED, result=Deferred@615296545, callback=, errback=) 00:27:37.280 [DEBUG - New I/O worker #933] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.280 [DEBUG - New I/O worker #944] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.281 [DEBUG - New I/O worker #817] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.281 [DEBUG - New I/O worker #817] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.281 [DEBUG - New I/O worker #817] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.281 [DEBUG - New I/O worker #817] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.282 [DEBUG - New I/O worker #817] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.282 [DEBUG - pool-3-thread-17] (Deferred.java:1330) callback=disconnect callback@1642712415 returned Deferred@1205975481(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1950875931 after disconnect callback@1642712415), errback=passthrough -> (continuation of Deferred@1950875931 after disconnect callback@1642712415)), so the following Deferred is getting paused: Deferred@1950875931(state=PAUSED, result=Deferred@1205975481, callback=, errback=) 00:27:37.283 [DEBUG - New I/O worker #818] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.283 [DEBUG - New I/O worker #862] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.284 [DEBUG - pool-3-thread-50] (Deferred.java:1330) callback=disconnect callback@705974995 returned Deferred@671097845(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1299429766 after disconnect callback@705974995), errback=passthrough -> (continuation of Deferred@1299429766 after disconnect callback@705974995)), so the following Deferred is getting paused: Deferred@1299429766(state=PAUSED, result=Deferred@671097845, callback=, errback=) 00:27:37.284 [DEBUG - New I/O worker #930] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.284 [DEBUG - New I/O worker #929] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.300 [DEBUG - New I/O worker #748] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.303 [DEBUG - New I/O worker #817] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.211 [DEBUG - New I/O worker #856] (Negotiator.java:764) Authenticated connection [id: 0x94f58957, /127.0.0.1:53346 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.211 [DEBUG - New I/O worker #897] (Negotiator.java:764) Authenticated connection [id: 0x3bc7317d, /127.0.0.1:53336 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.211 [DEBUG - New I/O worker #1196] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.210 [DEBUG - New I/O worker #945] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.210 [DEBUG - New I/O worker #855] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.210 [DEBUG - New I/O worker #1034] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.325 [DEBUG - New I/O worker #772] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.325 [DEBUG - New I/O worker #772] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1152), Traces: [0ms] refreshing cache from master, [59ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [59ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [60ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [462ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1078ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1143ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.326 [DEBUG - New I/O worker #848] (Negotiator.java:764) Authenticated connection [id: 0x2281b08d, /127.0.0.1:38084 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.326 [DEBUG - New I/O worker #772] (Deferred.java:1330) callback=retry RPC@383379046 returned Deferred@317087845(state=PENDING, result=null, callback=(continuation of Deferred@1474620509 after retry RPC@383379046), errback=(continuation of Deferred@1474620509 after retry RPC@383379046)), so the following Deferred is getting paused: Deferred@1474620509(state=PAUSED, result=Deferred@317087845, callback=org.apache.kudu.client.AsyncKuduClient$6@3fca0317 -> wakeup thread pool-3-thread-54, errback=passthrough -> wakeup thread pool-3-thread-54) 00:27:37.210 [DEBUG - New I/O worker #1038] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.200 [DEBUG - New I/O worker #1028] (Negotiator.java:764) Authenticated connection [id: 0xb9b10c7c, /127.0.0.1:53418 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.198 [DEBUG - New I/O worker #775] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.197 [DEBUG - New I/O worker #918] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.197 [DEBUG - New I/O worker #917] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.197 [DEBUG - pool-3-thread-68] (Deferred.java:1330) callback=disconnect callback@427105130 returned Deferred@307443412(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@959304507 after disconnect callback@427105130), errback=passthrough -> (continuation of Deferred@959304507 after disconnect callback@427105130)), so the following Deferred is getting paused: Deferred@959304507(state=PAUSED, result=Deferred@307443412, callback=, errback=) 00:27:37.188 [DEBUG - New I/O worker #812] (Negotiator.java:764) Authenticated connection [id: 0x5ad2d18c, /127.0.0.1:37952 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.184 [DEBUG - New I/O worker #863] (Negotiator.java:764) Authenticated connection [id: 0x241a023c, /127.0.0.1:53338 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.182 [DEBUG - New I/O worker #780] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.332 [DEBUG - New I/O worker #780] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1183), Traces: [0ms] refreshing cache from master, [97ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [98ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [98ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [413ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [524ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1033ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.332 [DEBUG - New I/O worker #780] (Deferred.java:1330) callback=retry RPC@1487824414 returned Deferred@1553070547(state=PENDING, result=null, callback=(continuation of Deferred@429091877 after retry RPC@1487824414), errback=(continuation of Deferred@429091877 after retry RPC@1487824414)), so the following Deferred is getting paused: Deferred@429091877(state=PAUSED, result=Deferred@1553070547, callback=org.apache.kudu.client.AsyncKuduClient$6@375093e -> wakeup thread pool-3-thread-63, errback=passthrough -> wakeup thread pool-3-thread-63) 00:27:37.174 [DEBUG - New I/O worker #846] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1071), Traces: [0ms] refreshing cache from master, [243ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [243ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [243ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [380ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [603ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [837ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.173 [DEBUG - New I/O worker #784] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299996, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299996, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299996, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.167 [DEBUG - New I/O worker #816] (Negotiator.java:764) Authenticated connection [id: 0xd19ddf4a, /127.0.0.1:37990 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.159 [DEBUG - New I/O worker #751] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=970), Traces: [0ms] refreshing cache from master, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [283ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [723ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [838ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.344 [DEBUG - New I/O worker #784] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.344 [DEBUG - New I/O worker #969] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.158 [DEBUG - New I/O worker #1024] (Negotiator.java:764) Authenticated connection [id: 0x6d0d2f63, /127.0.0.1:53388 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.153 [DEBUG - New I/O worker #854] (Negotiator.java:764) Authenticated connection [id: 0xa8731a19, /127.0.0.1:38000 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.352 [DEBUG - pool-3-thread-72] (Deferred.java:1330) callback=disconnect callback@153707023 returned Deferred@526520005(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1480164632 after disconnect callback@153707023), errback=passthrough -> (continuation of Deferred@1480164632 after disconnect callback@153707023)), so the following Deferred is getting paused: Deferred@1480164632(state=PAUSED, result=Deferred@526520005, callback=, errback=) 00:27:37.352 [DEBUG - New I/O worker #1161] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.152 [DEBUG - New I/O worker #789] (Negotiator.java:764) Authenticated connection [id: 0x8a96f84f, /127.0.0.1:37988 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.152 [DEBUG - New I/O worker #866] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=763), Traces: [0ms] refreshing cache from master, [474ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [474ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [474ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [540ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [632ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.355 [DEBUG - New I/O worker #866] (Deferred.java:1330) callback=retry RPC@1647047696 returned Deferred@1712614541(state=PENDING, result=null, callback=(continuation of Deferred@588709419 after retry RPC@1647047696), errback=(continuation of Deferred@588709419 after retry RPC@1647047696)), so the following Deferred is getting paused: Deferred@588709419(state=PAUSED, result=Deferred@1712614541, callback=org.apache.kudu.client.AsyncKuduClient$6@2fc9d65b -> wakeup thread pool-3-thread-73, errback=passthrough -> wakeup thread pool-3-thread-73) 00:27:37.151 [DEBUG - New I/O worker #1012] (Negotiator.java:764) Authenticated connection [id: 0x2c428e41, /127.0.0.1:39976 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.151 [DEBUG - New I/O worker #882] (Negotiator.java:764) Authenticated connection [id: 0x27313331, /127.0.0.1:53366 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.151 [DEBUG - New I/O worker #1027] (Negotiator.java:764) Authenticated connection [id: 0x163163f5, /127.0.0.1:53370 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.374 [DEBUG - New I/O worker #791] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.150 [DEBUG - New I/O worker #1026] (Negotiator.java:764) Authenticated connection [id: 0x7e4276f8, /127.0.0.1:40018 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.148 [DEBUG - New I/O worker #786] (Negotiator.java:764) Authenticated connection [id: 0xafc499e7, /127.0.0.1:38006 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.147 [DEBUG - New I/O worker #947] (Negotiator.java:764) Authenticated connection [id: 0x62fba10d, /127.0.0.1:39984 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.147 [DEBUG - New I/O worker #858] (Negotiator.java:764) Authenticated connection [id: 0x14bf6316, /127.0.0.1:53378 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.147 [DEBUG - New I/O worker #859] (Negotiator.java:764) Authenticated connection [id: 0xc9f585c4, /127.0.0.1:37970 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.143 [DEBUG - New I/O worker #847] (Deferred.java:1330) callback=retry RPC@1735740444 returned Deferred@1665848362(state=PENDING, result=null, callback=(continuation of Deferred@642270759 after retry RPC@1735740444), errback=(continuation of Deferred@642270759 after retry RPC@1735740444)), so the following Deferred is getting paused: Deferred@642270759(state=PAUSED, result=Deferred@1665848362, callback=org.apache.kudu.client.AsyncKuduClient$6@5afc75db -> wakeup thread pool-3-thread-48, errback=passthrough -> wakeup thread pool-3-thread-48) 00:27:37.142 [DEBUG - New I/O worker #931] (Negotiator.java:764) Authenticated connection [id: 0x21fbb742, /127.0.0.1:53360 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.138 [DEBUG - New I/O worker #1015] (Negotiator.java:764) Authenticated connection [id: 0x5fb05a2e, /127.0.0.1:39970 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.136 [DEBUG - New I/O worker #822] (Deferred.java:1330) callback=retry RPC@1419990098 returned Deferred@1352456734(state=PENDING, result=null, callback=(continuation of Deferred@362700393 after retry RPC@1419990098), errback=(continuation of Deferred@362700393 after retry RPC@1419990098)), so the following Deferred is getting paused: Deferred@362700393(state=PAUSED, result=Deferred@1352456734, callback=org.apache.kudu.client.AsyncKuduClient$6@6a8e7275 -> wakeup thread pool-3-thread-66, errback=passthrough -> wakeup thread pool-3-thread-66) 00:27:37.393 [DEBUG - New I/O worker #822] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.136 [DEBUG - New I/O worker #752] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1260), Traces: [0ms] refreshing cache from master, [347ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [349ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [349ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [654ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [692ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1213ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.393 [DEBUG - New I/O worker #752] (Deferred.java:1330) callback=retry RPC@1861681824 returned Deferred@1791531942(state=PENDING, result=null, callback=(continuation of Deferred@801769627 after retry RPC@1861681824), errback=(continuation of Deferred@801769627 after retry RPC@1861681824)), so the following Deferred is getting paused: Deferred@801769627(state=PAUSED, result=Deferred@1791531942, callback=org.apache.kudu.client.AsyncKuduClient$6@489bebf4 -> wakeup thread pool-3-thread-70, errback=passthrough -> wakeup thread pool-3-thread-70) 00:27:37.126 [DEBUG - New I/O worker #765] (Negotiator.java:764) Authenticated connection [id: 0x30705467, /127.0.0.1:38014 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.123 [DEBUG - New I/O worker #966] (Negotiator.java:764) Authenticated connection [id: 0x4b34c86a, /127.0.0.1:39968 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.123 [DEBUG - New I/O worker #799] (Negotiator.java:764) Authenticated connection [id: 0x371bf2aa, /127.0.0.1:38020 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.123 [DEBUG - New I/O worker #1017] (Negotiator.java:764) Authenticated connection [id: 0x76a039df, /127.0.0.1:53400 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.121 [DEBUG - New I/O worker #836] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.120 [DEBUG - New I/O worker #838] (Negotiator.java:764) Authenticated connection [id: 0x70873c48, /127.0.0.1:38024 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.115 [DEBUG - New I/O worker #1019] (Negotiator.java:764) Authenticated connection [id: 0x09bfd045, /127.0.0.1:40030 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.111 [DEBUG - New I/O worker #843] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.402 [DEBUG - New I/O worker #843] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1321), Traces: [0ms] refreshing cache from master, [253ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [253ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [253ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [462ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [537ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1013ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.402 [DEBUG - New I/O worker #765] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.402 [DEBUG - New I/O worker #765] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=644), Traces: [0ms] refreshing cache from master, [48ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [48ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [48ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [326ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [643ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [643ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.403 [DEBUG - New I/O worker #765] (Deferred.java:1330) callback=retry RPC@1630408721 returned Deferred@1695652076(state=PENDING, result=null, callback=(continuation of Deferred@538122794 after retry RPC@1630408721), errback=(continuation of Deferred@538122794 after retry RPC@1630408721)), so the following Deferred is getting paused: Deferred@538122794(state=PAUSED, result=Deferred@1695652076, callback=org.apache.kudu.client.AsyncKuduClient$6@c678a7 -> wakeup thread pool-3-thread-75, errback=passthrough -> wakeup thread pool-3-thread-75) 00:27:37.109 [DEBUG - New I/O worker #837] (Negotiator.java:764) Authenticated connection [id: 0xefcb5733, /127.0.0.1:38050 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.109 [DEBUG - New I/O worker #857] (Negotiator.java:764) Authenticated connection [id: 0x682fb485, /127.0.0.1:38032 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.109 [DEBUG - New I/O worker #1029] (Negotiator.java:764) Authenticated connection [id: 0xe38a77d8, /127.0.0.1:40008 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.109 [DEBUG - New I/O worker #1044] (Negotiator.java:764) Authenticated connection [id: 0x3711a8e8, /127.0.0.1:40020 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.105 [DEBUG - New I/O worker #1079] (Negotiator.java:764) Authenticated connection [id: 0xfbd710a1, /127.0.0.1:40002 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.104 [DEBUG - New I/O worker #1042] (Negotiator.java:764) Authenticated connection [id: 0x086f4d47, /127.0.0.1:40000 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.102 [DEBUG - New I/O worker #1025] (Negotiator.java:764) Authenticated connection [id: 0x96580ee9, /127.0.0.1:53386 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.099 [DEBUG - New I/O worker #865] (Negotiator.java:764) Authenticated connection [id: 0x9f137036, /127.0.0.1:38040 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.096 [DEBUG - New I/O worker #1046] (Negotiator.java:764) Authenticated connection [id: 0xbdeebca5, /127.0.0.1:40026 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.091 [DEBUG - New I/O worker #980] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.083 [DEBUG - New I/O worker #868] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.081 [DEBUG - New I/O worker #869] (Negotiator.java:764) Authenticated connection [id: 0x8e125f4a, /127.0.0.1:38008 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.067 [DEBUG - New I/O worker #749] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.409 [DEBUG - New I/O worker #749] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1203), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [695ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [821ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [887ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:37.409 [DEBUG - New I/O worker #749] (Deferred.java:1330) callback=retry RPC@2131372235 returned Deferred@2067117042(state=PENDING, result=null, callback=(continuation of Deferred@1043800816 after retry RPC@2131372235), errback=(continuation of Deferred@1043800816 after retry RPC@2131372235)), so the following Deferred is getting paused: Deferred@1043800816(state=PAUSED, result=Deferred@2067117042, callback=org.apache.kudu.client.AsyncKuduClient$6@2121f47c -> wakeup thread pool-3-thread-60, errback=passthrough -> wakeup thread pool-3-thread-60) 00:27:37.405 [DEBUG - New I/O worker #742] (Negotiator.java:764) Authenticated connection [id: 0x7b421b20, /127.0.0.1:38078 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.404 [DEBUG - New I/O worker #799] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.414 [DEBUG - New I/O worker #799] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1189), Traces: [0ms] refreshing cache from master, [582ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [582ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [586ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1157ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1179ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.414 [DEBUG - New I/O worker #799] (Deferred.java:1330) callback=retry RPC@1783279090 returned Deferred@1853174883(state=PENDING, result=null, callback=(continuation of Deferred@729262025 after retry RPC@1783279090), errback=(continuation of Deferred@729262025 after retry RPC@1783279090)), so the following Deferred is getting paused: Deferred@729262025(state=PAUSED, result=Deferred@1853174883, callback=org.apache.kudu.client.AsyncKuduClient$6@250c0073 -> wakeup thread pool-3-thread-29, errback=passthrough -> wakeup thread pool-3-thread-29) 00:27:37.402 [DEBUG - New I/O worker #752] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.402 [DEBUG - New I/O worker #843] (Deferred.java:1330) callback=retry RPC@588096301 returned Deferred@657601428(state=PENDING, result=null, callback=(continuation of Deferred@1647356182 after retry RPC@588096301), errback=(continuation of Deferred@1647356182 after retry RPC@588096301)), so the following Deferred is getting paused: Deferred@1647356182(state=PAUSED, result=Deferred@657601428, callback=org.apache.kudu.client.AsyncKuduClient$6@1d42c0cc -> wakeup thread pool-3-thread-38, errback=passthrough -> wakeup thread pool-3-thread-38) 00:27:37.416 [DEBUG - New I/O worker #869] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.416 [DEBUG - New I/O worker #869] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1044), Traces: [0ms] refreshing cache from master, [430ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [467ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [468ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1029ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1034ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1044ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.417 [DEBUG - New I/O worker #869] (Deferred.java:1330) callback=retry RPC@821922211 returned Deferred@885132132(state=PENDING, result=null, callback=(continuation of Deferred@1908442008 after retry RPC@821922211), errback=(continuation of Deferred@1908442008 after retry RPC@821922211)), so the following Deferred is getting paused: Deferred@1908442008(state=PAUSED, result=Deferred@885132132, callback=org.apache.kudu.client.AsyncKuduClient$6@1c977774 -> wakeup thread pool-3-thread-47, errback=passthrough -> wakeup thread pool-3-thread-47) 00:27:37.395 [DEBUG - New I/O worker #928] (Negotiator.java:764) Authenticated connection [id: 0x6e9fd7c0, /127.0.0.1:53424 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.394 [DEBUG - New I/O worker #1222] (Negotiator.java:764) Authenticated connection [id: 0x65a93b5d, /127.0.0.1:40066 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.393 [DEBUG - New I/O worker #786] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.419 [DEBUG - New I/O worker #786] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=720), Traces: [0ms] refreshing cache from master, [103ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [104ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [104ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [519ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [694ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [696ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:37.392 [DEBUG - New I/O worker #847] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.425 [DEBUG - New I/O worker #742] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.425 [DEBUG - New I/O worker #786] (Deferred.java:1330) callback=retry RPC@105890324 returned Deferred@40912389(state=PENDING, result=null, callback=(continuation of Deferred@1198704687 after retry RPC@105890324), errback=(continuation of Deferred@1198704687 after retry RPC@105890324)), so the following Deferred is getting paused: Deferred@1198704687(state=PAUSED, result=Deferred@40912389, callback=org.apache.kudu.client.AsyncKuduClient$6@e58a631 -> wakeup thread pool-3-thread-8, errback=passthrough -> wakeup thread pool-3-thread-8) 00:27:37.426 [DEBUG - New I/O worker #786] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.426 [DEBUG - New I/O worker #822] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.426 [DEBUG - New I/O worker #822] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.427 [DEBUG - New I/O worker #822] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.427 [DEBUG - New I/O worker #822] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.427 [DEBUG - New I/O worker #822] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.428 [DEBUG - pool-3-thread-66] (Deferred.java:1330) callback=disconnect callback@1140436343 returned Deferred@1500072447(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2105057945 after disconnect callback@1140436343), errback=passthrough -> (continuation of Deferred@2105057945 after disconnect callback@1140436343)), so the following Deferred is getting paused: Deferred@2105057945(state=PAUSED, result=Deferred@1500072447, callback=, errback=) 00:27:37.429 [DEBUG - New I/O worker #865] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.429 [DEBUG - New I/O worker #865] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1059), Traces: [0ms] refreshing cache from master, [455ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [456ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [456ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [977ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1012ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1059ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.429 [DEBUG - New I/O worker #865] (Deferred.java:1330) callback=retry RPC@812372343 returned Deferred@877941666(state=PENDING, result=null, callback=(continuation of Deferred@1901517644 after retry RPC@812372343), errback=(continuation of Deferred@1901517644 after retry RPC@812372343)), so the following Deferred is getting paused: Deferred@1901517644(state=PAUSED, result=Deferred@877941666, callback=org.apache.kudu.client.AsyncKuduClient$6@2ee3171c -> wakeup thread pool-3-thread-9, errback=passthrough -> wakeup thread pool-3-thread-9) 00:27:37.430 [DEBUG - New I/O worker #1032] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.430 [DEBUG - New I/O worker #838] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.430 [DEBUG - New I/O worker #838] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1065), Traces: [0ms] refreshing cache from master, [446ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [446ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [447ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1016ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1065ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.431 [DEBUG - New I/O worker #838] (Deferred.java:1330) callback=retry RPC@672193690 returned Deferred@741301568(state=PENDING, result=null, callback=(continuation of Deferred@1764611745 after retry RPC@672193690), errback=(continuation of Deferred@1764611745 after retry RPC@672193690)), so the following Deferred is getting paused: Deferred@1764611745(state=PAUSED, result=Deferred@741301568, callback=org.apache.kudu.client.AsyncKuduClient$6@76540b02 -> wakeup thread pool-3-thread-19, errback=passthrough -> wakeup thread pool-3-thread-19) 00:27:37.431 [DEBUG - New I/O worker #822] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.431 [DEBUG - New I/O worker #765] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.433 [DEBUG - New I/O worker #857] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.433 [DEBUG - New I/O worker #857] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1065), Traces: [0ms] refreshing cache from master, [450ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [451ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [451ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1017ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1064ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.433 [DEBUG - New I/O worker #857] (Deferred.java:1330) callback=retry RPC@1832918785 returned Deferred@1769969265(state=PENDING, result=null, callback=(continuation of Deferred@746394938 after retry RPC@1832918785), errback=(continuation of Deferred@746394938 after retry RPC@1832918785)), so the following Deferred is getting paused: Deferred@746394938(state=PAUSED, result=Deferred@1769969265, callback=org.apache.kudu.client.AsyncKuduClient$6@70ce8a41 -> wakeup thread pool-3-thread-7, errback=passthrough -> wakeup thread pool-3-thread-7) 00:27:37.434 [DEBUG - New I/O worker #837] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.435 [DEBUG - New I/O worker #837] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1066), Traces: [0ms] refreshing cache from master, [463ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [463ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [463ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [701ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1066ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.435 [DEBUG - New I/O worker #837] (Deferred.java:1330) callback=retry RPC@172985577 returned Deferred@242227576(state=PENDING, result=null, callback=(continuation of Deferred@1265796818 after retry RPC@172985577), errback=(continuation of Deferred@1265796818 after retry RPC@172985577)), so the following Deferred is getting paused: Deferred@1265796818(state=PAUSED, result=Deferred@242227576, callback=org.apache.kudu.client.AsyncKuduClient$6@c2db586 -> wakeup thread pool-3-thread-76, errback=passthrough -> wakeup thread pool-3-thread-76) 00:27:37.436 [DEBUG - New I/O worker #843] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.391 [DEBUG - New I/O worker #1196] (Negotiator.java:764) Authenticated connection [id: 0xb2abeded, /127.0.0.1:40060 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.391 [DEBUG - New I/O worker #1038] (Negotiator.java:764) Authenticated connection [id: 0xf3259e7e, /127.0.0.1:40072 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.390 [DEBUG - New I/O worker #1034] (Negotiator.java:764) Authenticated connection [id: 0x651dfb2f, /127.0.0.1:53442 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.390 [DEBUG - New I/O worker #945] (Negotiator.java:764) Authenticated connection [id: 0xd573a339, /127.0.0.1:40054 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.387 [DEBUG - New I/O worker #855] (Negotiator.java:764) Authenticated connection [id: 0x0418a3aa, /127.0.0.1:38090 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.387 [DEBUG - New I/O worker #859] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.442 [DEBUG - New I/O worker #859] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1140), Traces: [0ms] refreshing cache from master, [131ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [131ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [132ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [221ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [309ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1085ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.442 [DEBUG - New I/O worker #752] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.383 [DEBUG - New I/O worker #1057] (Negotiator.java:764) Authenticated connection [id: 0x53cd718a, /127.0.0.1:53430 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.443 [DEBUG - New I/O worker #847] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.443 [DEBUG - New I/O worker #847] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.443 [DEBUG - New I/O worker #847] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.444 [DEBUG - New I/O worker #847] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.444 [DEBUG - New I/O worker #847] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.454 [DEBUG - pool-3-thread-48] (Deferred.java:1330) callback=disconnect callback@703096435 returned Deferred@1319997935(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@964470898 after disconnect callback@703096435), errback=passthrough -> (continuation of Deferred@964470898 after disconnect callback@703096435)), so the following Deferred is getting paused: Deferred@964470898(state=PAUSED, result=Deferred@1319997935, callback=, errback=) 00:27:37.454 [DEBUG - New I/O worker #1061] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.454 [DEBUG - New I/O worker #1062] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.377 [DEBUG - New I/O worker #1036] (Negotiator.java:764) Authenticated connection [id: 0x691ede06, /127.0.0.1:53436 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.377 [DEBUG - New I/O worker #744] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.377 [DEBUG - New I/O worker #852] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.375 [DEBUG - New I/O worker #741] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.375 [DEBUG - New I/O worker #820] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.374 [DEBUG - New I/O worker #848] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.374 [DEBUG - New I/O worker #1022] (Negotiator.java:764) Authenticated connection [id: 0x18bcdc2d, /127.0.0.1:40042 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.373 [DEBUG - New I/O worker #854] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.459 [DEBUG - New I/O worker #854] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1086), Traces: [0ms] refreshing cache from master, [423ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [424ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [424ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [943ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [997ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [1001ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:37.459 [DEBUG - New I/O worker #854] (Deferred.java:1330) callback=retry RPC@1488533214 returned Deferred@1552328977(state=PENDING, result=null, callback=(continuation of Deferred@428096741 after retry RPC@1488533214), errback=(continuation of Deferred@428096741 after retry RPC@1488533214)), so the following Deferred is getting paused: Deferred@428096741(state=PAUSED, result=Deferred@1552328977, callback=org.apache.kudu.client.AsyncKuduClient$6@3949887e -> wakeup thread pool-3-thread-15, errback=passthrough -> wakeup thread pool-3-thread-15) 00:27:37.373 [DEBUG - New I/O worker #866] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.372 [DEBUG - New I/O worker #789] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.460 [DEBUG - New I/O worker #789] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=819), Traces: [0ms] refreshing cache from master, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [675ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [730ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [761ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:37.461 [DEBUG - New I/O worker #865] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.461 [DEBUG - New I/O worker #789] (Deferred.java:1330) callback=retry RPC@22559264 returned Deferred@90681043(state=PENDING, result=null, callback=(continuation of Deferred@1080373275 after retry RPC@22559264), errback=(continuation of Deferred@1080373275 after retry RPC@22559264)), so the following Deferred is getting paused: Deferred@1080373275(state=PAUSED, result=Deferred@90681043, callback=org.apache.kudu.client.AsyncKuduClient$6@31264769 -> wakeup thread pool-3-thread-27, errback=passthrough -> wakeup thread pool-3-thread-27) 00:27:37.461 [DEBUG - New I/O worker #838] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.462 [DEBUG - New I/O worker #749] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.463 [DEBUG - New I/O worker #837] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.463 [DEBUG - New I/O worker #854] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.369 [DEBUG - New I/O worker #1030] (Negotiator.java:764) Authenticated connection [id: 0x1ffc6fdf, /127.0.0.1:40048 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.363 [DEBUG - New I/O worker #841] (Negotiator.java:764) Authenticated connection [id: 0xfe19ca89, /127.0.0.1:38096 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.363 [DEBUG - New I/O worker #929] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.356 [DEBUG - New I/O worker #839] (Negotiator.java:764) Authenticated connection [id: 0xddbef21b, /127.0.0.1:38038 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.350 [DEBUG - New I/O worker #816] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.465 [DEBUG - New I/O worker #816] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=928), Traces: [0ms] refreshing cache from master, [245ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [245ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [245ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [798ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [812ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [863ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:37.348 [DEBUG - New I/O worker #812] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.468 [DEBUG - New I/O worker #812] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1107), Traces: [0ms] refreshing cache from master, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [102ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [335ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [974ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.468 [DEBUG - New I/O worker #812] (Deferred.java:1330) callback=retry RPC@558527059 returned Deferred@628482722(state=PENDING, result=null, callback=(continuation of Deferred@1618442344 after retry RPC@558527059), errback=(continuation of Deferred@1618442344 after retry RPC@558527059)), so the following Deferred is getting paused: Deferred@1618442344(state=PAUSED, result=Deferred@628482722, callback=org.apache.kudu.client.AsyncKuduClient$6@23f7bead -> wakeup thread pool-3-thread-34, errback=passthrough -> wakeup thread pool-3-thread-34) 00:27:37.468 [DEBUG - New I/O worker #816] (Deferred.java:1330) callback=retry RPC@35091481 returned Deferred@103346307(state=PENDING, result=null, callback=(continuation of Deferred@1126858274 after retry RPC@35091481), errback=(continuation of Deferred@1126858274 after retry RPC@35091481)), so the following Deferred is getting paused: Deferred@1126858274(state=PAUSED, result=Deferred@103346307, callback=org.apache.kudu.client.AsyncKuduClient$6@4b2d7ab7 -> wakeup thread pool-3-thread-59, errback=passthrough -> wakeup thread pool-3-thread-59) 00:27:37.347 [DEBUG - New I/O worker #748] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.344 [DEBUG - New I/O worker #751] (Deferred.java:1330) callback=retry RPC@1597295934 returned Deferred@1527468544(state=PENDING, result=null, callback=(continuation of Deferred@503957253 after retry RPC@1597295934), errback=(continuation of Deferred@503957253 after retry RPC@1597295934)), so the following Deferred is getting paused: Deferred@503957253(state=PAUSED, result=Deferred@1527468544, callback=org.apache.kudu.client.AsyncKuduClient$6@56f9550d -> wakeup thread pool-3-thread-67, errback=passthrough -> wakeup thread pool-3-thread-67) 00:27:37.470 [DEBUG - New I/O worker #751] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.342 [DEBUG - New I/O worker #846] (Deferred.java:1330) callback=retry RPC@1437758481 returned Deferred@1368255103(state=PENDING, result=null, callback=(continuation of Deferred@344948266 after retry RPC@1437758481), errback=(continuation of Deferred@344948266 after retry RPC@1437758481)), so the following Deferred is getting paused: Deferred@344948266(state=PAUSED, result=Deferred@1368255103, callback=org.apache.kudu.client.AsyncKuduClient$6@6305e84d -> wakeup thread pool-3-thread-23, errback=passthrough -> wakeup thread pool-3-thread-23) 00:27:37.470 [DEBUG - New I/O worker #846] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.341 [DEBUG - New I/O worker #823] (Negotiator.java:764) Authenticated connection [id: 0x0f058462, /127.0.0.1:38066 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.336 [DEBUG - New I/O worker #780] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.335 [DEBUG - New I/O worker #772] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.257 [DEBUG - New I/O worker #739] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.470 [DEBUG - New I/O worker #839] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.474 [DEBUG - New I/O worker #839] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1107), Traces: [0ms] refreshing cache from master, [458ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [459ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [459ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [1047ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1071ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1104ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.474 [DEBUG - New I/O worker #839] (Deferred.java:1330) callback=retry RPC@1673468318 returned Deferred@1736479025(state=PENDING, result=null, callback=(continuation of Deferred@578950053 after retry RPC@1673468318), errback=(continuation of Deferred@578950053 after retry RPC@1673468318)), so the following Deferred is getting paused: Deferred@578950053(state=PAUSED, result=Deferred@1736479025, callback=org.apache.kudu.client.AsyncKuduClient$6@8024f46 -> wakeup thread pool-3-thread-41, errback=passthrough -> wakeup thread pool-3-thread-41) 00:27:37.475 [DEBUG - New I/O worker #823] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.475 [DEBUG - New I/O worker #823] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1105), Traces: [0ms] refreshing cache from master, [478ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [478ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [478ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [796ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1095ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1105ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.476 [DEBUG - New I/O worker #823] (Deferred.java:1330) callback=retry RPC@2091645121 returned Deferred@2022994828(state=PENDING, result=null, callback=(continuation of Deferred@1033302778 after retry RPC@2091645121), errback=(continuation of Deferred@1033302778 after retry RPC@2091645121)), so the following Deferred is getting paused: Deferred@1033302778(state=PAUSED, result=Deferred@2022994828, callback=org.apache.kudu.client.AsyncKuduClient$6@2fafa64c -> wakeup thread pool-3-thread-52, errback=passthrough -> wakeup thread pool-3-thread-52) 00:27:37.466 [DEBUG - New I/O worker #799] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.460 [DEBUG - New I/O worker #855] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.476 [DEBUG - New I/O worker #855] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=590), Traces: [0ms] refreshing cache from master, [306ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [306ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [306ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [548ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [566ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [573ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:37.477 [DEBUG - New I/O worker #855] (Deferred.java:1330) callback=retry RPC@854303093 returned Deferred@919876592(state=PENDING, result=null, callback=(continuation of Deferred@1943448398 after retry RPC@854303093), errback=(continuation of Deferred@1943448398 after retry RPC@854303093)), so the following Deferred is getting paused: Deferred@1943448398(state=PAUSED, result=Deferred@919876592, callback=org.apache.kudu.client.AsyncKuduClient$6@2d6005ec -> wakeup thread pool-3-thread-1, errback=passthrough -> wakeup thread pool-3-thread-1) 00:27:37.460 [DEBUG - New I/O worker #857] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.478 [DEBUG - New I/O worker #772] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.479 [DEBUG - New I/O worker #772] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.481 [DEBUG - New I/O worker #772] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.481 [DEBUG - New I/O worker #772] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.459 [DEBUG - New I/O worker #843] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.482 [DEBUG - New I/O worker #843] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.483 [DEBUG - New I/O worker #843] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.483 [DEBUG - New I/O worker #843] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.483 [DEBUG - New I/O worker #843] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.459 [DEBUG - New I/O worker #848] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=575), Traces: [0ms] refreshing cache from master, [307ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [308ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [309ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [447ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [557ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [570ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:37.484 [DEBUG - New I/O worker #848] (Deferred.java:1330) callback=retry RPC@1808468548 returned Deferred@1878295525(state=PENDING, result=null, callback=(continuation of Deferred@720765055 after retry RPC@1808468548), errback=(continuation of Deferred@720765055 after retry RPC@1808468548)), so the following Deferred is getting paused: Deferred@720765055(state=PAUSED, result=Deferred@1878295525, callback=org.apache.kudu.client.AsyncKuduClient$6@283c0df4 -> wakeup thread pool-3-thread-49, errback=passthrough -> wakeup thread pool-3-thread-49) 00:27:37.456 [DEBUG - New I/O worker #847] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.485 [DEBUG - New I/O worker #772] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.485 [DEBUG - New I/O worker #839] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.485 [DEBUG - pool-3-thread-54] (Deferred.java:1330) callback=disconnect callback@1179754310 returned Deferred@1933212688(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@408156101 after disconnect callback@1179754310), errback=passthrough -> (continuation of Deferred@408156101 after disconnect callback@1179754310)), so the following Deferred is getting paused: Deferred@408156101(state=PAUSED, result=Deferred@1933212688, callback=, errback=) 00:27:37.486 [DEBUG - New I/O worker #913] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.486 [DEBUG - New I/O worker #854] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.486 [DEBUG - New I/O worker #854] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.487 [DEBUG - New I/O worker #772] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.451 [DEBUG - New I/O worker #786] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.488 [DEBUG - New I/O worker #786] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.488 [DEBUG - New I/O worker #786] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.488 [DEBUG - New I/O worker #786] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.488 [DEBUG - New I/O worker #786] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.489 [DEBUG - New I/O worker #848] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.449 [DEBUG - New I/O worker #765] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.490 [DEBUG - New I/O worker #765] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.490 [DEBUG - New I/O worker #765] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.492 [DEBUG - New I/O worker #765] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.492 [DEBUG - New I/O worker #765] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.442 [DEBUG - New I/O worker #752] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.442 [DEBUG - New I/O worker #859] (Deferred.java:1330) callback=retry RPC@1939558401 returned Deferred@2007292586(state=PENDING, result=null, callback=(continuation of Deferred@849762874 after retry RPC@1939558401), errback=(continuation of Deferred@849762874 after retry RPC@1939558401)), so the following Deferred is getting paused: Deferred@849762874(state=PAUSED, result=Deferred@2007292586, callback=org.apache.kudu.client.AsyncKuduClient$6@2a6cefd5 -> wakeup thread pool-3-thread-36, errback=passthrough -> wakeup thread pool-3-thread-36) 00:27:37.493 [DEBUG - New I/O worker #859] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.494 [DEBUG - pool-3-thread-75] (Deferred.java:1330) callback=disconnect callback@1247757664 returned Deferred@1326093572(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1502092681 after disconnect callback@1247757664), errback=passthrough -> (continuation of Deferred@1502092681 after disconnect callback@1247757664)), so the following Deferred is getting paused: Deferred@1502092681(state=PAUSED, result=Deferred@1326093572, callback=, errback=) 00:27:37.494 [DEBUG - New I/O worker #932] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.494 [DEBUG - New I/O worker #931] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.430 [DEBUG - New I/O worker #1040] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.426 [DEBUG - New I/O worker #742] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=389), Traces: [0ms] refreshing cache from master, [147ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [147ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [147ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [387ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.495 [DEBUG - New I/O worker #742] (Deferred.java:1330) callback=retry RPC@920199758 returned Deferred@853975117(state=PENDING, result=null, callback=(continuation of Deferred@2011441269 after retry RPC@920199758), errback=(continuation of Deferred@2011441269 after retry RPC@920199758)), so the following Deferred is getting paused: Deferred@2011441269(state=PAUSED, result=Deferred@853975117, callback=org.apache.kudu.client.AsyncKuduClient$6@39ed4297 -> wakeup thread pool-3-thread-57, errback=passthrough -> wakeup thread pool-3-thread-57) 00:27:37.422 [DEBUG - New I/O worker #869] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.497 [DEBUG - New I/O worker #848] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.498 [DEBUG - New I/O worker #848] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.498 [DEBUG - New I/O worker #848] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.498 [DEBUG - New I/O worker #848] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.497 [DEBUG - New I/O worker #843] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.496 [DEBUG - New I/O worker #816] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.500 [DEBUG - New I/O worker #855] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.496 [DEBUG - New I/O worker #866] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.496 [DEBUG - New I/O worker #780] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.503 [DEBUG - New I/O worker #780] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.503 [DEBUG - New I/O worker #780] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.496 [DEBUG - New I/O worker #1190] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.496 [DEBUG - New I/O worker #1013] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.495 [DEBUG - pool-3-thread-38] (Deferred.java:1330) callback=disconnect callback@1536433835 returned Deferred@516797697(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@296320904 after disconnect callback@1536433835), errback=passthrough -> (continuation of Deferred@296320904 after disconnect callback@1536433835)), so the following Deferred is getting paused: Deferred@296320904(state=PAUSED, result=Deferred@516797697, callback=, errback=) 00:27:37.495 [DEBUG - New I/O worker #765] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.493 [DEBUG - New I/O worker #752] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.506 [DEBUG - New I/O worker #752] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.507 [DEBUG - New I/O worker #752] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.492 [DEBUG - New I/O worker #854] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.508 [DEBUG - New I/O worker #854] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.508 [DEBUG - New I/O worker #854] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.491 [DEBUG - New I/O worker #789] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.509 [DEBUG - pool-3-thread-15] (Deferred.java:1330) callback=disconnect callback@1940856878 returned Deferred@715189340(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@4854189 after disconnect callback@1940856878), errback=passthrough -> (continuation of Deferred@4854189 after disconnect callback@1940856878)), so the following Deferred is getting paused: Deferred@4854189(state=PAUSED, result=Deferred@715189340, callback=, errback=) 00:27:37.491 [DEBUG - New I/O worker #865] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.490 [DEBUG - New I/O worker #839] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.486 [DEBUG - New I/O worker #914] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.484 [DEBUG - New I/O worker #751] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.522 [DEBUG - New I/O worker #751] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.523 [DEBUG - New I/O worker #751] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.523 [DEBUG - New I/O worker #751] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.523 [DEBUG - New I/O worker #751] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.524 [DEBUG - pool-3-thread-67] (Deferred.java:1330) callback=disconnect callback@432731842 returned Deferred@1476972662(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1818099823 after disconnect callback@432731842), errback=passthrough -> (continuation of Deferred@1818099823 after disconnect callback@432731842)), so the following Deferred is getting paused: Deferred@1818099823(state=PAUSED, result=Deferred@1476972662, callback=, errback=) 00:27:37.524 [DEBUG - New I/O worker #936] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.525 [DEBUG - New I/O worker #943] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.480 [DEBUG - New I/O worker #823] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.474 [DEBUG - New I/O worker #836] (Negotiator.java:764) Authenticated connection [id: 0xaa7b5dcf, /127.0.0.1:38072 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.524 [DEBUG - pool-3-thread-31] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.523 [DEBUG - New I/O worker #865] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.527 [DEBUG - New I/O worker #865] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.527 [DEBUG - New I/O worker #865] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.527 [DEBUG - New I/O worker #865] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.528 [DEBUG - pool-3-thread-9] (Deferred.java:1330) callback=disconnect callback@1063211800 returned Deferred@1548997756(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1194422925 after disconnect callback@1063211800), errback=passthrough -> (continuation of Deferred@1194422925 after disconnect callback@1063211800)), so the following Deferred is getting paused: Deferred@1194422925(state=PAUSED, result=Deferred@1548997756, callback=, errback=) 00:27:37.528 [DEBUG - New I/O worker #1024] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.529 [DEBUG - New I/O worker #1026] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.530 [DEBUG - New I/O worker #841] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.530 [DEBUG - New I/O worker #841] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=648), Traces: [0ms] refreshing cache from master, [319ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [321ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [321ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [562ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [577ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [648ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.531 [DEBUG - New I/O worker #865] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.531 [DEBUG - New I/O worker #841] (Deferred.java:1330) callback=retry RPC@2008564459 returned Deferred@1938278596(state=PENDING, result=null, callback=(continuation of Deferred@914704592 after retry RPC@2008564459), errback=(continuation of Deferred@914704592 after retry RPC@2008564459)), so the following Deferred is getting paused: Deferred@914704592(state=PAUSED, result=Deferred@1938278596, callback=org.apache.kudu.client.AsyncKuduClient$6@32e32fb7 -> wakeup thread pool-3-thread-51, errback=passthrough -> wakeup thread pool-3-thread-51) 00:27:37.522 [DEBUG - New I/O worker #859] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.531 [DEBUG - New I/O worker #859] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.532 [DEBUG - New I/O worker #859] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.532 [DEBUG - New I/O worker #789] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.532 [DEBUG - New I/O worker #789] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.532 [DEBUG - New I/O worker #789] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.532 [DEBUG - New I/O worker #789] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.533 [DEBUG - New I/O worker #789] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.522 [DEBUG - New I/O worker #855] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.533 [DEBUG - New I/O worker #855] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.534 [DEBUG - New I/O worker #855] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.534 [DEBUG - pool-3-thread-35] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.534 [DEBUG - New I/O worker #897] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.522 [DEBUG - New I/O worker #756] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.522 [DEBUG - New I/O worker #839] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.535 [DEBUG - New I/O worker #789] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.521 [DEBUG - New I/O worker #816] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.535 [DEBUG - New I/O worker #816] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.536 [DEBUG - New I/O worker #816] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.536 [DEBUG - New I/O worker #816] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.536 [DEBUG - New I/O worker #816] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.537 [DEBUG - New I/O worker #863] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.538 [DEBUG - New I/O worker #816] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.515 [DEBUG - pool-3-thread-78] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.515 [DEBUG - New I/O worker #752] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.515 [DEBUG - pool-3-thread-70] (Deferred.java:1330) callback=disconnect callback@226204172 returned Deferred@1060965821(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@690944878 after disconnect callback@226204172), errback=passthrough -> (continuation of Deferred@690944878 after disconnect callback@226204172)), so the following Deferred is getting paused: Deferred@690944878(state=PAUSED, result=Deferred@1060965821, callback=, errback=) 00:27:37.515 [DEBUG - New I/O worker #1012] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.514 [DEBUG - New I/O worker #869] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.540 [DEBUG - New I/O worker #869] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.540 [DEBUG - New I/O worker #869] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.540 [DEBUG - New I/O worker #869] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.540 [DEBUG - New I/O worker #869] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.514 [DEBUG - New I/O worker #854] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.512 [DEBUG - New I/O worker #755] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.542 [DEBUG - pool-3-thread-47] (Deferred.java:1330) callback=disconnect callback@1332772007 returned Deferred@1552925740(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1488575096 after disconnect callback@1332772007), errback=passthrough -> (continuation of Deferred@1488575096 after disconnect callback@1332772007)), so the following Deferred is getting paused: Deferred@1488575096(state=PAUSED, result=Deferred@1552925740, callback=, errback=) 00:27:37.542 [DEBUG - New I/O worker #1017] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.543 [DEBUG - New I/O worker #1019] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.511 [DEBUG - New I/O worker #856] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.504 [DEBUG - New I/O worker #780] (NetUtil.java:106) Resolved IP of `127.22.19.195' to /127.22.19.195 in 583019ns 00:27:37.545 [DEBUG - New I/O worker #780] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.504 [DEBUG - pool-3-thread-24] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.503 [DEBUG - New I/O worker #846] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.502 [DEBUG - New I/O worker #866] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.502 [DEBUG - New I/O worker #742] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.546 [DEBUG - New I/O worker #749] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.547 [DEBUG - New I/O worker #749] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.502 [DEBUG - New I/O worker #848] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299996, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299996, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299996, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.547 [DEBUG - New I/O worker #1382] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.548 [DEBUG - pool-3-thread-49] (Deferred.java:1330) callback=disconnect callback@1272090230 returned Deferred@1527305533(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1362608764 after disconnect callback@1272090230), errback=passthrough -> (continuation of Deferred@1362608764 after disconnect callback@1272090230)), so the following Deferred is getting paused: Deferred@1362608764(state=PAUSED, result=Deferred@1527305533, callback=, errback=) 00:27:37.548 [DEBUG - New I/O worker #1057] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.549 [DEBUG - New I/O worker #1196] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.550 [DEBUG - New I/O worker #836] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.550 [DEBUG - New I/O worker #836] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=668), Traces: [0ms] refreshing cache from master, [184ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [184ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [184ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [459ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [587ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [669ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.551 [DEBUG - New I/O worker #848] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.551 [DEBUG - New I/O worker #841] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.552 [DEBUG - New I/O worker #837] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.552 [DEBUG - New I/O worker #780] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.553 [DEBUG - New I/O worker #948] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.553 [DEBUG - New I/O worker #1086] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.553 [DEBUG - pool-3-thread-63] (Deferred.java:1330) callback=disconnect callback@1728365318 returned Deferred@266875185(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@884586941 after disconnect callback@1728365318), errback=passthrough -> (continuation of Deferred@884586941 after disconnect callback@1728365318)), so the following Deferred is getting paused: Deferred@884586941(state=PAUSED, result=Deferred@266875185, callback=, errback=) 00:27:37.546 [DEBUG - New I/O worker #1339] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.546 [DEBUG - New I/O worker #857] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.555 [DEBUG - New I/O worker #857] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.555 [DEBUG - New I/O worker #857] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.556 [DEBUG - New I/O worker #857] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.556 [DEBUG - New I/O worker #857] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.544 [DEBUG - New I/O worker #1348] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.541 [DEBUG - New I/O worker #812] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.540 [DEBUG - New I/O worker #799] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.561 [DEBUG - New I/O worker #799] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.561 [DEBUG - New I/O worker #799] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.561 [DEBUG - New I/O worker #799] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.562 [DEBUG - New I/O worker #799] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.540 [DEBUG - New I/O worker #739] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.563 [DEBUG - New I/O worker #739] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.563 [DEBUG - New I/O worker #739] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.566 [DEBUG - New I/O worker #739] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.566 [DEBUG - New I/O worker #739] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.567 [DEBUG - pool-3-thread-3] (Deferred.java:1330) callback=disconnect callback@656401197 returned Deferred@62037899(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1911614678 after disconnect callback@656401197), errback=passthrough -> (continuation of Deferred@1911614678 after disconnect callback@656401197)), so the following Deferred is getting paused: Deferred@1911614678(state=PAUSED, result=Deferred@62037899, callback=, errback=) 00:27:37.567 [DEBUG - New I/O worker #745] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.537 [DEBUG - pool-3-thread-59] (Deferred.java:1330) callback=disconnect callback@454315728 returned Deferred@762087023(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@110727608 after disconnect callback@454315728), errback=passthrough -> (continuation of Deferred@110727608 after disconnect callback@454315728)), so the following Deferred is getting paused: Deferred@110727608(state=PAUSED, result=Deferred@762087023, callback=, errback=) 00:27:37.537 [DEBUG - New I/O worker #1015] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.568 [DEBUG - New I/O worker #743] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.535 [DEBUG - New I/O worker #839] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.569 [DEBUG - New I/O worker #839] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.569 [DEBUG - New I/O worker #839] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.534 [DEBUG - New I/O worker #966] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.534 [DEBUG - pool-3-thread-27] (Deferred.java:1330) callback=disconnect callback@557775373 returned Deferred@1043689074(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@899984406 after disconnect callback@557775373), errback=passthrough -> (continuation of Deferred@899984406 after disconnect callback@557775373)), so the following Deferred is getting paused: Deferred@899984406(state=PAUSED, result=Deferred@1043689074, callback=, errback=) 00:27:37.534 [DEBUG - New I/O worker #855] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.570 [DEBUG - New I/O worker #855] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.532 [DEBUG - New I/O worker #859] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.571 [DEBUG - New I/O worker #859] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.572 [DEBUG - pool-3-thread-36] (Deferred.java:1330) callback=disconnect callback@1090010034 returned Deferred@1733237925(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1001599117 after disconnect callback@1090010034), errback=passthrough -> (continuation of Deferred@1001599117 after disconnect callback@1090010034)), so the following Deferred is getting paused: Deferred@1001599117(state=PAUSED, result=Deferred@1733237925, callback=, errback=) 00:27:37.572 [DEBUG - New I/O worker #1072] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.573 [DEBUG - New I/O worker #1170] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.573 [DEBUG - New I/O worker #749] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.573 [DEBUG - New I/O worker #749] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.574 [DEBUG - New I/O worker #749] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.574 [DEBUG - pool-3-thread-60] (Deferred.java:1330) callback=disconnect callback@1961315876 returned Deferred@1108577790(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@477898981 after disconnect callback@1961315876), errback=passthrough -> (continuation of Deferred@477898981 after disconnect callback@1961315876)), so the following Deferred is getting paused: Deferred@477898981(state=PAUSED, result=Deferred@1108577790, callback=, errback=) 00:27:37.575 [DEBUG - New I/O worker #753] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.530 [DEBUG - New I/O worker #823] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.575 [DEBUG - New I/O worker #823] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.575 [DEBUG - New I/O worker #823] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.576 [DEBUG - New I/O worker #823] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.576 [DEBUG - New I/O worker #823] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.530 [DEBUG - New I/O worker #838] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.576 [DEBUG - New I/O worker #838] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.577 [DEBUG - New I/O worker #838] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.577 [DEBUG - New I/O worker #838] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.577 [DEBUG - New I/O worker #838] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.579 [DEBUG - pool-3-thread-52] (Deferred.java:1330) callback=disconnect callback@425825890 returned Deferred@174435935(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1084400047 after disconnect callback@425825890), errback=passthrough -> (continuation of Deferred@1084400047 after disconnect callback@425825890)), so the following Deferred is getting paused: Deferred@1084400047(state=PAUSED, result=Deferred@174435935, callback=, errback=) 00:27:37.527 [DEBUG - New I/O worker #751] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.579 [DEBUG - New I/O worker #780] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.579 [DEBUG - pool-3-thread-19] (Deferred.java:1330) callback=disconnect callback@1521790401 returned Deferred@1955006139(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1298077472 after disconnect callback@1521790401), errback=passthrough -> (continuation of Deferred@1298077472 after disconnect callback@1521790401)), so the following Deferred is getting paused: Deferred@1298077472(state=PAUSED, result=Deferred@1955006139, callback=, errback=) 00:27:37.579 [DEBUG - New I/O worker #1020] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.580 [DEBUG - New I/O worker #1022] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.574 [DEBUG - New I/O worker #754] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.580 [DEBUG - New I/O worker #1027] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.581 [DEBUG - New I/O worker #1042] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.572 [DEBUG - New I/O worker #1222] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.571 [DEBUG - New I/O worker #1036] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.571 [DEBUG - pool-3-thread-1] (Deferred.java:1330) callback=disconnect callback@1691131938 returned Deferred@1768626013(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@174894843 after disconnect callback@1691131938), errback=passthrough -> (continuation of Deferred@174894843 after disconnect callback@1691131938)), so the following Deferred is getting paused: Deferred@174894843(state=PAUSED, result=Deferred@1768626013, callback=, errback=) 00:27:37.570 [DEBUG - New I/O worker #1044] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.570 [DEBUG - pool-3-thread-41] (Deferred.java:1330) callback=disconnect callback@2042917953 returned Deferred@921462580(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@58819842 after disconnect callback@2042917953), errback=passthrough -> (continuation of Deferred@58819842 after disconnect callback@2042917953)), so the following Deferred is getting paused: Deferred@58819842(state=PAUSED, result=Deferred@921462580, callback=, errback=) 00:27:37.570 [DEBUG - New I/O worker #1025] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.565 [DEBUG - New I/O worker #1029] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.565 [DEBUG - New I/O worker #858] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.584 [DEBUG - New I/O worker #799] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.584 [DEBUG - New I/O worker #869] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.584 [DEBUG - New I/O worker #739] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.586 [DEBUG - New I/O worker #855] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.565 [DEBUG - pool-3-thread-7] (Deferred.java:1330) callback=disconnect callback@1948797061 returned Deferred@1485122111(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@358008588 after disconnect callback@1948797061), errback=passthrough -> (continuation of Deferred@358008588 after disconnect callback@1948797061)), so the following Deferred is getting paused: Deferred@358008588(state=PAUSED, result=Deferred@1485122111, callback=, errback=) 00:27:37.564 [DEBUG - New I/O worker #1079] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.564 [DEBUG - New I/O worker #882] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.563 [DEBUG - pool-3-thread-29] (Deferred.java:1330) callback=disconnect callback@923646340 returned Deferred@99747253(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1575113348 after disconnect callback@923646340), errback=passthrough -> (continuation of Deferred@1575113348 after disconnect callback@923646340)), so the following Deferred is getting paused: Deferred@1575113348(state=PAUSED, result=Deferred@99747253, callback=, errback=) 00:27:37.563 [DEBUG - New I/O worker #841] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.587 [DEBUG - New I/O worker #841] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.587 [DEBUG - New I/O worker #841] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.587 [DEBUG - New I/O worker #841] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.552 [DEBUG - New I/O worker #837] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.667 [DEBUG - New I/O worker #837] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.667 [DEBUG - New I/O worker #837] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.667 [DEBUG - New I/O worker #837] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.668 [DEBUG - New I/O worker #1161] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.669 [DEBUG - New I/O worker #1046] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.551 [DEBUG - New I/O worker #836] (Deferred.java:1330) callback=retry RPC@1472513840 returned Deferred@1408981264(state=PENDING, result=null, callback=(continuation of Deferred@385469707 after retry RPC@1472513840), errback=(continuation of Deferred@385469707 after retry RPC@1472513840)), so the following Deferred is getting paused: Deferred@385469707(state=PAUSED, result=Deferred@1408981264, callback=org.apache.kudu.client.AsyncKuduClient$6@56297d86 -> wakeup thread pool-3-thread-44, errback=passthrough -> wakeup thread pool-3-thread-44) 00:27:37.670 [DEBUG - New I/O worker #836] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.670 [DEBUG - pool-3-thread-43] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.550 [DEBUG - New I/O worker #846] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.671 [DEBUG - New I/O worker #846] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.671 [DEBUG - New I/O worker #846] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.671 [DEBUG - New I/O worker #846] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.674 [DEBUG - New I/O worker #836] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.674 [DEBUG - New I/O worker #836] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.674 [DEBUG - New I/O worker #836] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.675 [DEBUG - New I/O worker #836] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.675 [DEBUG - New I/O worker #837] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.675 [DEBUG - pool-3-thread-23] (Deferred.java:1330) callback=disconnect callback@34107140 returned Deferred@907152012(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1026821982 after disconnect callback@34107140), errback=passthrough -> (continuation of Deferred@1026821982 after disconnect callback@34107140)), so the following Deferred is getting paused: Deferred@1026821982(state=PAUSED, result=Deferred@907152012, callback=, errback=) 00:27:37.675 [DEBUG - New I/O worker #1382] (Negotiator.java:764) Authenticated connection [id: 0x23092b1d, /127.0.0.1:40078 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.676 [DEBUG - New I/O worker #1065] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.676 [DEBUG - New I/O worker #1339] (Negotiator.java:764) Authenticated connection [id: 0x75249f2c, /127.0.0.1:38102 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.678 [DEBUG - New I/O worker #751] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.550 [DEBUG - New I/O worker #866] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.678 [DEBUG - pool-3-thread-77] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.676 [DEBUG - New I/O worker #817] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.676 [DEBUG - New I/O worker #1063] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.676 [DEBUG - New I/O worker #836] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.680 [DEBUG - New I/O worker #1339] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.669 [DEBUG - New I/O worker #775] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.680 [DEBUG - pool-3-thread-44] (Deferred.java:1330) callback=disconnect callback@1576225962 returned Deferred@2087078478(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@706013045 after disconnect callback@1576225962), errback=passthrough -> (continuation of Deferred@706013045 after disconnect callback@1576225962)), so the following Deferred is getting paused: Deferred@706013045(state=PAUSED, result=Deferred@2087078478, callback=, errback=) 00:27:37.680 [DEBUG - New I/O worker #1339] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=146), Traces: [0ms] refreshing cache from master, [7ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [7ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [7ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [145ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [146ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:37.681 [DEBUG - New I/O worker #1030] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.668 [DEBUG - New I/O worker #1023] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.668 [DEBUG - pool-3-thread-76] (Deferred.java:1330) callback=disconnect callback@2007093288 returned Deferred@1006685933(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@778412638 after disconnect callback@2007093288), errback=passthrough -> (continuation of Deferred@778412638 after disconnect callback@2007093288)), so the following Deferred is getting paused: Deferred@778412638(state=PAUSED, result=Deferred@1006685933, callback=, errback=) 00:27:37.682 [DEBUG - New I/O worker #836] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.667 [DEBUG - New I/O worker #1348] (Negotiator.java:764) Authenticated connection [id: 0x2c6f3ae9, /127.0.0.1:53448 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.666 [DEBUG - New I/O worker #822] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.665 [DEBUG - New I/O worker #855] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.684 [DEBUG - New I/O worker #1339] (Deferred.java:1330) callback=retry RPC@1619620817 returned Deferred@1689713432(state=PENDING, result=null, callback=(continuation of Deferred@565475818 after retry RPC@1619620817), errback=(continuation of Deferred@565475818 after retry RPC@1619620817)), so the following Deferred is getting paused: Deferred@565475818(state=PAUSED, result=Deferred@1689713432, callback=org.apache.kudu.client.AsyncKuduClient$6@3be36064 -> wakeup thread pool-3-thread-35, errback=passthrough -> wakeup thread pool-3-thread-35) 00:27:37.663 [DEBUG - New I/O worker #742] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.684 [DEBUG - New I/O worker #742] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.684 [DEBUG - New I/O worker #742] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.685 [DEBUG - New I/O worker #742] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.685 [DEBUG - New I/O worker #742] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.686 [DEBUG - pool-3-thread-57] (Deferred.java:1330) callback=disconnect callback@1030528663 returned Deferred@1753575789(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1993931015 after disconnect callback@1030528663), errback=passthrough -> (continuation of Deferred@1993931015 after disconnect callback@1030528663)), so the following Deferred is getting paused: Deferred@1993931015(state=PAUSED, result=Deferred@1753575789, callback=, errback=) 00:27:37.686 [DEBUG - New I/O worker #945] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.663 [DEBUG - New I/O worker #812] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.687 [DEBUG - New I/O worker #812] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.687 [DEBUG - New I/O worker #812] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.687 [DEBUG - New I/O worker #812] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.688 [DEBUG - New I/O worker #812] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.662 [DEBUG - New I/O worker #869] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.688 [DEBUG - pool-3-thread-34] (Deferred.java:1330) callback=disconnect callback@126180314 returned Deferred@1339873957(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@709027878 after disconnect callback@126180314), errback=passthrough -> (continuation of Deferred@709027878 after disconnect callback@126180314)), so the following Deferred is getting paused: Deferred@709027878(state=PAUSED, result=Deferred@1339873957, callback=, errback=) 00:27:37.662 [DEBUG - pool-3-thread-39] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.661 [DEBUG - New I/O worker #780] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.661 [DEBUG - New I/O worker #848] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.661 [DEBUG - New I/O worker #856] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.661 [DEBUG - New I/O worker #755] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.661 [DEBUG - New I/O worker #1331] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.690 [DEBUG - New I/O worker #812] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.661 [DEBUG - New I/O worker #1356] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.661 [DEBUG - New I/O worker #816] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.660 [DEBUG - New I/O worker #865] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.660 [DEBUG - New I/O worker #1342] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.660 [DEBUG - New I/O worker #947] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.660 [DEBUG - New I/O worker #1357] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.660 [DEBUG - pool-3-thread-8] (Deferred.java:1330) callback=disconnect callback@1294678196 returned Deferred@1900819314(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2144763377 after disconnect callback@1294678196), errback=passthrough -> (continuation of Deferred@2144763377 after disconnect callback@1294678196)), so the following Deferred is getting paused: Deferred@2144763377(state=PAUSED, result=Deferred@1900819314, callback=, errback=) 00:27:37.660 [DEBUG - New I/O worker #1416] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.696 [DEBUG - New I/O worker #1432] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.660 [DEBUG - New I/O worker #1406] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.660 [DEBUG - New I/O worker #1415] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.659 [DEBUG - New I/O worker #1404] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.659 [DEBUG - New I/O worker #1324] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.659 [DEBUG - New I/O worker #919] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.659 [DEBUG - New I/O worker #786] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.659 [DEBUG - New I/O worker #765] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.659 [DEBUG - New I/O worker #847] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.704 [DEBUG - New I/O worker #1356] (Negotiator.java:764) Authenticated connection [id: 0x00344f15, /127.0.0.1:53454 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.659 [DEBUG - New I/O worker #843] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.659 [DEBUG - New I/O worker #841] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.590 [DEBUG - New I/O worker #823] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.705 [DEBUG - pool-3-thread-51] (Deferred.java:1330) callback=disconnect callback@575422009 returned Deferred@308105065(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@12525349 after disconnect callback@575422009), errback=passthrough -> (continuation of Deferred@12525349 after disconnect callback@575422009)), so the following Deferred is getting paused: Deferred@12525349(state=PAUSED, result=Deferred@308105065, callback=, errback=) 00:27:37.589 [DEBUG - New I/O worker #749] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.706 [DEBUG - New I/O worker #1034] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.589 [DEBUG - New I/O worker #838] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.588 [DEBUG - New I/O worker #859] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.584 [DEBUG - New I/O worker #857] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.581 [DEBUG - New I/O worker #839] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.707 [DEBUG - New I/O worker #841] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.707 [DEBUG - New I/O worker #1038] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.703 [DEBUG - New I/O worker #1431] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.689 [DEBUG - New I/O worker #742] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.689 [DEBUG - New I/O worker #1327] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.689 [DEBUG - New I/O worker #1033] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.688 [DEBUG - New I/O worker #1031] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.686 [DEBUG - New I/O worker #928] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.722 [DEBUG - New I/O worker #1431] (Negotiator.java:764) Authenticated connection [id: 0xdf36ec6b, /127.0.0.1:53472 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.735 [DEBUG - New I/O worker #1404] (Negotiator.java:764) Authenticated connection [id: 0x5a202f63, /127.0.0.1:53466 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.740 [DEBUG - New I/O worker #1324] (Negotiator.java:764) Authenticated connection [id: 0x0ce9a164, /127.0.0.1:38120 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.741 [DEBUG - New I/O worker #1416] (Negotiator.java:764) Authenticated connection [id: 0x77a5379f, /127.0.0.1:40090 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.742 [DEBUG - New I/O worker #1415] (Negotiator.java:764) Authenticated connection [id: 0x52bb480f, /127.0.0.1:53460 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.743 [DEBUG - New I/O worker #1324] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.743 [DEBUG - New I/O worker #1324] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=217), Traces: [0ms] refreshing cache from master, [96ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [96ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [96ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [215ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [217ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.744 [DEBUG - New I/O worker #1324] (Deferred.java:1330) callback=retry RPC@26617858 returned Deferred@95010024(state=PENDING, result=null, callback=(continuation of Deferred@1084957241 after retry RPC@26617858), errback=(continuation of Deferred@1084957241 after retry RPC@26617858)), so the following Deferred is getting paused: Deferred@1084957241(state=PAUSED, result=Deferred@95010024, callback=org.apache.kudu.client.AsyncKuduClient$6@41c7c0a3 -> wakeup thread pool-3-thread-31, errback=passthrough -> wakeup thread pool-3-thread-31) 00:27:37.746 [DEBUG - New I/O worker #1324] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.749 [DEBUG - New I/O worker #1324] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.749 [DEBUG - New I/O worker #1324] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.749 [DEBUG - New I/O worker #1324] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.749 [DEBUG - New I/O worker #1324] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.750 [DEBUG - New I/O worker #1324] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.750 [DEBUG - New I/O worker #1432] (Negotiator.java:764) Authenticated connection [id: 0xdcbd195c, /127.0.0.1:40102 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.751 [DEBUG - New I/O worker #1404] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.752 [DEBUG - New I/O worker #1324] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.684 [DEBUG - New I/O worker #1339] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.680 [DEBUG - New I/O worker #1028] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.679 [DEBUG - New I/O worker #846] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.678 [DEBUG - New I/O worker #866] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.753 [DEBUG - New I/O worker #866] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.754 [DEBUG - New I/O worker #866] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.754 [DEBUG - New I/O worker #1049] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:37.754171 24969 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:37.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.624130 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40096 00:27:37.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.624549 (+ 419us) server_negotiation.cc:184] Beginning negotiation 00:27:37.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.624572 (+ 23us) server_negotiation.cc:373] Waiting for connection header 00:27:37.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.624611 (+ 39us) server_negotiation.cc:381] Connection header received 00:27:37.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.624718 (+ 107us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:37.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.624744 (+ 26us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:37.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.624859 (+ 115us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:37.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.625013 (+ 154us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:37.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.698965 (+ 73952us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:37.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.700176 (+ 1211us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:37.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.722479 (+ 22303us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:37.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.723780 (+ 1301us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:37.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.723960 (+ 180us) server_negotiation.cc:589] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 00:27:37.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.751793 (+ 27833us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40096: BlockingRecv error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) 00:27:37.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":152} 00:27:37.754 [DEBUG - pool-3-thread-73] (Deferred.java:1330) callback=disconnect callback@1762891348 returned Deferred@309860951(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1844631038 after disconnect callback@1762891348), errback=passthrough -> (continuation of Deferred@1844631038 after disconnect callback@1762891348)), so the following Deferred is getting paused: Deferred@1844631038(state=PAUSED, result=Deferred@309860951, callback=, errback=) 00:27:37.755 [DEBUG - New I/O worker #1204] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.756 [DEBUG - New I/O worker #1327] (Negotiator.java:764) Authenticated connection [id: 0x99f0be39, /127.0.0.1:38126 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.756 [DEBUG - New I/O worker #1357] (Negotiator.java:764) Authenticated connection [id: 0xe68b01ec, /127.0.0.1:40084 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.759 [DEBUG - New I/O worker #1331] (Negotiator.java:764) Authenticated connection [id: 0x9763bf59, /127.0.0.1:38108 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.759 [DEBUG - New I/O worker #1327] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.759 [DEBUG - New I/O worker #1327] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=88), Traces: [0ms] refreshing cache from master, [11ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [11ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [11ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [84ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [89ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.760 [DEBUG - New I/O worker #1342] (Negotiator.java:764) Authenticated connection [id: 0x8db4a4f4, /127.0.0.1:38114 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.760 [DEBUG - New I/O worker #1327] (Deferred.java:1330) callback=retry RPC@1186031898 returned Deferred@1116665108(state=PENDING, result=null, callback=(continuation of Deferred@126640929 after retry RPC@1186031898), errback=(continuation of Deferred@126640929 after retry RPC@1186031898)), so the following Deferred is getting paused: Deferred@126640929(state=PAUSED, result=Deferred@1116665108, callback=org.apache.kudu.client.AsyncKuduClient$6@7bf5480a -> wakeup thread pool-3-thread-43, errback=passthrough -> wakeup thread pool-3-thread-43) 00:27:37.761 [DEBUG - New I/O worker #1339] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.761 [DEBUG - New I/O worker #1339] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.761 [DEBUG - New I/O worker #1339] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.761 [DEBUG - New I/O worker #1339] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.762 [DEBUG - New I/O worker #1331] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.762 [DEBUG - New I/O worker #1331] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=212), Traces: [0ms] refreshing cache from master, [62ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [65ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [67ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [209ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [211ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [211ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:37.762 [DEBUG - New I/O worker #1331] (Deferred.java:1330) callback=retry RPC@1925185245 returned Deferred@1988125778(state=PENDING, result=null, callback=(continuation of Deferred@864220390 after retry RPC@1925185245), errback=(continuation of Deferred@864220390 after retry RPC@1925185245)), so the following Deferred is getting paused: Deferred@864220390(state=PAUSED, result=Deferred@1988125778, callback=org.apache.kudu.client.AsyncKuduClient$6@f3ffcab -> wakeup thread pool-3-thread-24, errback=passthrough -> wakeup thread pool-3-thread-24) 00:27:37.762 [DEBUG - New I/O worker #1339] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.762 [DEBUG - New I/O worker #1327] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.762 [DEBUG - New I/O worker #1342] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.763 [DEBUG - pool-3-thread-35] (Deferred.java:1330) callback=disconnect callback@1698430177 returned Deferred@558040420(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@319041537 after disconnect callback@1698430177), errback=passthrough -> (continuation of Deferred@319041537 after disconnect callback@1698430177)), so the following Deferred is getting paused: Deferred@319041537(state=PAUSED, result=Deferred@558040420, callback=, errback=) 00:27:37.764 [DEBUG - New I/O worker #1348] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.765 [DEBUG - New I/O worker #1331] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.765 [DEBUG - New I/O worker #1382] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.765 [DEBUG - New I/O worker #1339] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.763 [DEBUG - New I/O worker #1342] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=224), Traces: [0ms] refreshing cache from master, [81ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [81ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [82ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [206ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [223ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [224ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.766 [DEBUG - New I/O worker #1342] (Deferred.java:1330) callback=retry RPC@794910567 returned Deferred@727633491(state=PENDING, result=null, callback=(continuation of Deferred@1851544924 after retry RPC@794910567), errback=(continuation of Deferred@1851544924 after retry RPC@794910567)), so the following Deferred is getting paused: Deferred@1851544924(state=PAUSED, result=Deferred@727633491, callback=org.apache.kudu.client.AsyncKuduClient$6@7ac77a09 -> wakeup thread pool-3-thread-78, errback=passthrough -> wakeup thread pool-3-thread-78) 00:27:37.767 [DEBUG - New I/O worker #1327] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.767 [DEBUG - New I/O worker #1327] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.767 [DEBUG - New I/O worker #1327] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.768 [DEBUG - New I/O worker #1327] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.768 [DEBUG - New I/O worker #1327] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.770 [DEBUG - pool-3-thread-61] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.770 [DEBUG - New I/O worker #1331] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.770 [DEBUG - pool-3-thread-43] (Deferred.java:1330) callback=disconnect callback@2144064276 returned Deferred@54215174(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1513176948 after disconnect callback@2144064276), errback=passthrough -> (continuation of Deferred@1513176948 after disconnect callback@2144064276)), so the following Deferred is getting paused: Deferred@1513176948(state=PAUSED, result=Deferred@54215174, callback=, errback=) 00:27:37.770 [DEBUG - New I/O worker #1342] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.770 [DEBUG - New I/O worker #1331] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.771 [DEBUG - New I/O worker #1431] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.772 [DEBUG - New I/O worker #1327] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.771 [DEBUG - New I/O worker #1432] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.771 [DEBUG - New I/O worker #1331] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.772 [DEBUG - New I/O worker #1331] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.772 [DEBUG - New I/O worker #1331] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.773 [DEBUG - New I/O worker #799] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.773 [DEBUG - New I/O worker #739] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.773 [DEBUG - pool-3-thread-24] (Deferred.java:1330) callback=disconnect callback@2129419752 returned Deferred@1718314433(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1928030218 after disconnect callback@2129419752), errback=passthrough -> (continuation of Deferred@1928030218 after disconnect callback@2129419752)), so the following Deferred is getting paused: Deferred@1928030218(state=PAUSED, result=Deferred@1718314433, callback=, errback=) 00:27:37.774 [DEBUG - pool-3-thread-58] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.774 [DEBUG - New I/O worker #1357] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.775 [DEBUG - New I/O worker #1356] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.775 [DEBUG - New I/O worker #1342] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.775 [DEBUG - New I/O worker #1342] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.775 [DEBUG - New I/O worker #1342] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.775 [DEBUG - New I/O worker #1342] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.776 [DEBUG - New I/O worker #1342] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.777 [DEBUG - New I/O worker #1331] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.777 [DEBUG - pool-3-thread-78] (Deferred.java:1330) callback=disconnect callback@1378998948 returned Deferred@1962731123(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@711948807 after disconnect callback@1378998948), errback=passthrough -> (continuation of Deferred@711948807 after disconnect callback@1378998948)), so the following Deferred is getting paused: Deferred@711948807(state=PAUSED, result=Deferred@1962731123, callback=, errback=) 00:27:37.778 [DEBUG - New I/O worker #1485] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.778 [DEBUG - New I/O worker #1415] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.778 [DEBUG - New I/O worker #1390] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.778 [DEBUG - New I/O worker #1416] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.779 [DEBUG - New I/O worker #1342] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.781 [DEBUG - New I/O worker #1353] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.783 [DEBUG - pool-3-thread-10] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.790 [DEBUG - New I/O worker #1390] (Negotiator.java:764) Authenticated connection [id: 0x12213a32, /127.0.0.1:53478 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.790 [DEBUG - New I/O worker #1485] (Negotiator.java:764) Authenticated connection [id: 0xb4613ea8, /127.0.0.1:40108 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.794 [DEBUG - New I/O worker #1353] (Negotiator.java:764) Authenticated connection [id: 0x9fc1c6d8, /127.0.0.1:38132 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.795 [DEBUG - pool-3-thread-11] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.797 [DEBUG - New I/O worker #1353] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.797 [DEBUG - New I/O worker #1353] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=22), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [19ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [19ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [22ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.797 [DEBUG - New I/O worker #772] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.799 [DEBUG - New I/O worker #812] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.799 [DEBUG - New I/O worker #1353] (Deferred.java:1330) callback=retry RPC@1334400237 returned Deferred@1270274500(state=PENDING, result=null, callback=(continuation of Deferred@246701782 after retry RPC@1334400237), errback=(continuation of Deferred@246701782 after retry RPC@1334400237)), so the following Deferred is getting paused: Deferred@246701782(state=PAUSED, result=Deferred@1270274500, callback=org.apache.kudu.client.AsyncKuduClient$6@3adb96a6 -> wakeup thread pool-3-thread-58, errback=passthrough -> wakeup thread pool-3-thread-58) 00:27:37.800 [DEBUG - New I/O worker #1468] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.801 [DEBUG - New I/O worker #1353] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.800 [DEBUG - New I/O worker #1394] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.801 [DEBUG - New I/O worker #1467] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.806 [DEBUG - New I/O worker #1353] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.807 [DEBUG - New I/O worker #1353] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.807 [DEBUG - New I/O worker #1353] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.807 [DEBUG - New I/O worker #1353] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.807 [DEBUG - New I/O worker #1353] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.808 [DEBUG - pool-3-thread-58] (Deferred.java:1330) callback=disconnect callback@1718696898 returned Deferred@442764910(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@484904990 after disconnect callback@1718696898), errback=passthrough -> (continuation of Deferred@484904990 after disconnect callback@1718696898)), so the following Deferred is getting paused: Deferred@484904990(state=PAUSED, result=Deferred@442764910, callback=, errback=) 00:27:37.809 [DEBUG - New I/O worker #1485] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.810 [DEBUG - New I/O worker #1353] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.811 [DEBUG - New I/O worker #1390] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.813 [DEBUG - New I/O worker #1468] (Negotiator.java:764) Authenticated connection [id: 0x48e587ae, /127.0.0.1:40114 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.820 [DEBUG - New I/O worker #1394] (Negotiator.java:764) Authenticated connection [id: 0xfd627515, /127.0.0.1:38138 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.822 [DEBUG - New I/O worker #1394] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.822 [DEBUG - New I/O worker #1394] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=27), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [21ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [27ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.822 [DEBUG - New I/O worker #1394] (Deferred.java:1330) callback=retry RPC@1192729177 returned Deferred@1126696547(state=PENDING, result=null, callback=(continuation of Deferred@103452770 after retry RPC@1192729177), errback=(continuation of Deferred@103452770 after retry RPC@1192729177)), so the following Deferred is getting paused: Deferred@103452770(state=PAUSED, result=Deferred@1126696547, callback=org.apache.kudu.client.AsyncKuduClient$6@4c58cd41 -> wakeup thread pool-3-thread-11, errback=passthrough -> wakeup thread pool-3-thread-11) 00:27:37.824 [DEBUG - New I/O worker #1394] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.827 [DEBUG - New I/O worker #1394] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.827 [DEBUG - New I/O worker #1394] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.827 [DEBUG - New I/O worker #1394] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.827 [DEBUG - New I/O worker #1394] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.827 [DEBUG - New I/O worker #1394] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.828 [DEBUG - New I/O worker #1394] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:37.834764 25003 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.797774 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53484 00:27:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.798217 (+ 443us) server_negotiation.cc:184] Beginning negotiation 00:27:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.798243 (+ 26us) server_negotiation.cc:373] Waiting for connection header 00:27:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.801074 (+ 2831us) server_negotiation.cc:381] Connection header received 00:27:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.801215 (+ 141us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.801242 (+ 27us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.801413 (+ 171us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.831438 (+ 30025us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.831864 (+ 426us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.833215 (+ 1351us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.833707 (+ 492us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53484: BlockingWrite error: write error: Broken pipe (error 32) 00:27:37.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":137} 00:27:37.835 [DEBUG - New I/O worker #789] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.845 [DEBUG - pool-3-thread-21] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.848 [DEBUG - pool-3-thread-37] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.850 [DEBUG - Time-limited test] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.850 [DEBUG - pool-3-thread-64] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.851 [DEBUG - pool-3-thread-14] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.852 [DEBUG - New I/O worker #1342] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.852 [DEBUG - New I/O worker #1331] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.853 [DEBUG - pool-3-thread-62] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.856 [DEBUG - pool-3-thread-65] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.859 [DEBUG - New I/O worker #1401] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.859 [DEBUG - New I/O worker #1327] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.860 [DEBUG - New I/O worker #1352] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.861 [DEBUG - New I/O worker #1339] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.862 [DEBUG - New I/O worker #1204] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.861 [DEBUG - New I/O worker #1381] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.862 [DEBUG - New I/O worker #846] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.875 [DEBUG - New I/O worker #1406] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:37.875 [DEBUG - pool-3-thread-11] (Deferred.java:1330) callback=disconnect callback@915338697 returned Deferred@1845763070(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@953909400 after disconnect callback@915338697), errback=passthrough -> (continuation of Deferred@953909400 after disconnect callback@915338697)), so the following Deferred is getting paused: Deferred@953909400(state=PAUSED, result=Deferred@1845763070, callback=, errback=) 00:27:37.875 [INFO - New I/O worker #1406] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:37.876 [DEBUG - New I/O worker #1467] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:37.876 [DEBUG - pool-3-thread-31] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.876 [INFO - New I/O worker #1467] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:37.876 [DEBUG - New I/O worker #1468] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.876 [DEBUG - New I/O worker #742] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.876 [DEBUG - New I/O worker #839] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.877 [DEBUG - New I/O worker #857] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.877 [DEBUG - New I/O worker #1468] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.877 [DEBUG - New I/O worker #859] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.878 [DEBUG - New I/O worker #838] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.878 [DEBUG - New I/O worker #749] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.879 [DEBUG - New I/O worker #823] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.880 [DEBUG - New I/O worker #947] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.887 [DEBUG - New I/O worker #836] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.888 [DEBUG - New I/O worker #1395] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.890 [DEBUG - New I/O worker #1451] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.890 [DEBUG - New I/O worker #1449] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.895 [DEBUG - New I/O worker #837] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.908 [DEBUG - New I/O worker #1395] (Negotiator.java:764) Authenticated connection [id: 0xb23ad67e, /127.0.0.1:38150 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.912 [DEBUG - New I/O worker #1451] (Negotiator.java:764) Authenticated connection [id: 0x9034d223, /127.0.0.1:40126 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.913 [DEBUG - New I/O worker #1347] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.913 [DEBUG - New I/O worker #1424] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.915 [DEBUG - New I/O worker #1395] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.916 [DEBUG - New I/O worker #1395] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=227), Traces: [0ms] refreshing cache from master, [197ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [197ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [197ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [225ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [227ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:37.915 [DEBUG - New I/O worker #1423] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.915 [DEBUG - New I/O worker #1449] (Negotiator.java:764) Authenticated connection [id: 0xcdc650c6, /127.0.0.1:53496 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.915 [DEBUG - New I/O worker #841] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:37.921 [DEBUG - New I/O worker #1395] (Deferred.java:1330) callback=retry RPC@2040777490 returned Deferred@2107396350(state=PENDING, result=null, callback=(continuation of Deferred@949928233 after retry RPC@2040777490), errback=(continuation of Deferred@949928233 after retry RPC@2040777490)), so the following Deferred is getting paused: Deferred@949928233(state=PAUSED, result=Deferred@2107396350, callback=org.apache.kudu.client.AsyncKuduClient$6@6e7025ff -> wakeup thread pool-3-thread-39, errback=passthrough -> wakeup thread pool-3-thread-39) 00:27:37.922 [DEBUG - New I/O worker #1395] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.925 [DEBUG - New I/O worker #1395] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.926 [DEBUG - New I/O worker #1395] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.926 [DEBUG - New I/O worker #1395] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.926 [DEBUG - New I/O worker #1395] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.926 [DEBUG - New I/O worker #1395] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.927 [DEBUG - New I/O worker #1347] (Negotiator.java:764) Authenticated connection [id: 0x5869aff5, /127.0.0.1:38156 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.927 [DEBUG - pool-3-thread-39] (Deferred.java:1330) callback=disconnect callback@1512575335 returned Deferred@1912032082(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1048384396 after disconnect callback@1512575335), errback=passthrough -> (continuation of Deferred@1048384396 after disconnect callback@1512575335)), so the following Deferred is getting paused: Deferred@1048384396(state=PAUSED, result=Deferred@1912032082, callback=, errback=) 00:27:37.928 [DEBUG - New I/O worker #1451] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.929 [DEBUG - New I/O worker #1449] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.929 [DEBUG - New I/O worker #1395] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.931 [DEBUG - New I/O worker #1424] (Negotiator.java:764) Authenticated connection [id: 0x95de22f3, /127.0.0.1:40132 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.938 [DEBUG - pool-3-thread-28] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.941 [DEBUG - New I/O worker #1423] (Negotiator.java:764) Authenticated connection [id: 0x0d60e3c3, /127.0.0.1:53502 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.943 [DEBUG - pool-3-thread-42] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.947 [DEBUG - New I/O worker #1347] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.947 [DEBUG - New I/O worker #1347] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=268), Traces: [0ms] refreshing cache from master, [230ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [230ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [230ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [250ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [255ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [264ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:37.950 [DEBUG - New I/O worker #1347] (Deferred.java:1330) callback=retry RPC@863438674 returned Deferred@927503844(state=PENDING, result=null, callback=(continuation of Deferred@1917455721 after retry RPC@863438674), errback=(continuation of Deferred@1917455721 after retry RPC@863438674)), so the following Deferred is getting paused: Deferred@1917455721(state=PAUSED, result=Deferred@927503844, callback=org.apache.kudu.client.AsyncKuduClient$6@48b409ea -> wakeup thread pool-3-thread-77, errback=passthrough -> wakeup thread pool-3-thread-77) 00:27:37.951 [DEBUG - New I/O worker #1347] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:37.952 [DEBUG - pool-3-thread-30] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.955 [DEBUG - New I/O worker #1347] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:37.959 [DEBUG - New I/O worker #1347] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:37.959 [DEBUG - New I/O worker #1347] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:37.960 [DEBUG - New I/O worker #1347] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:37.960 [DEBUG - New I/O worker #1347] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:37.960 [DEBUG - pool-3-thread-80] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.961 [DEBUG - pool-3-thread-77] (Deferred.java:1330) callback=disconnect callback@1509062039 returned Deferred@940515963(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1411369485 after disconnect callback@1509062039), errback=passthrough -> (continuation of Deferred@1411369485 after disconnect callback@1509062039)), so the following Deferred is getting paused: Deferred@1411369485(state=PAUSED, result=Deferred@940515963, callback=, errback=) 00:27:37.962 [DEBUG - New I/O worker #1423] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:37.962 [DEBUG - pool-3-thread-2] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.963 [DEBUG - New I/O worker #1424] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:37.964 [DEBUG - New I/O worker #1347] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:37.971 [DEBUG - New I/O worker #1474] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:37.972 [DEBUG - New I/O worker #1475] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:37.976 [DEBUG - New I/O worker #1397] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:37.988 [DEBUG - New I/O worker #1475] (Negotiator.java:764) Authenticated connection [id: 0x424dcb9f, /127.0.0.1:40138 => /127.22.19.253:37879] using SASL/PLAIN 00:27:37.993 [DEBUG - New I/O worker #1352] (Negotiator.java:764) Authenticated connection [id: 0x075a13c8, /127.0.0.1:38144 => /127.22.19.252:32833] using SASL/PLAIN 00:27:37.996 [DEBUG - New I/O worker #1352] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:37.996 [DEBUG - New I/O worker #1352] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=213), Traces: [0ms] refreshing cache from master, [71ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [71ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [72ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [213ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:37.998 [DEBUG - pool-3-thread-4] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:37.998 [DEBUG - New I/O worker #1474] (Negotiator.java:764) Authenticated connection [id: 0x84f6009e, /127.0.0.1:53508 => /127.22.19.254:44849] using SASL/PLAIN 00:27:37.999 [DEBUG - New I/O worker #1352] (Deferred.java:1330) callback=retry RPC@1313106566 returned Deferred@1249638294(state=PENDING, result=null, callback=(continuation of Deferred@259612861 after retry RPC@1313106566), errback=(continuation of Deferred@259612861 after retry RPC@1313106566)), so the following Deferred is getting paused: Deferred@259612861(state=PAUSED, result=Deferred@1249638294, callback=org.apache.kudu.client.AsyncKuduClient$6@5341b918 -> wakeup thread pool-3-thread-10, errback=passthrough -> wakeup thread pool-3-thread-10) 00:27:38.000 [DEBUG - New I/O worker #1352] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.005 [DEBUG - New I/O worker #1352] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.005 [DEBUG - New I/O worker #1352] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.006 [DEBUG - New I/O worker #1352] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.006 [DEBUG - New I/O worker #1352] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.006 [DEBUG - New I/O worker #1352] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:38.008801 25003 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.855269 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53490 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.855797 (+ 528us) server_negotiation.cc:184] Beginning negotiation 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.855823 (+ 26us) server_negotiation.cc:373] Waiting for connection header 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.861127 (+ 5304us) server_negotiation.cc:381] Connection header received 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.861278 (+ 151us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.861305 (+ 27us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.861453 (+ 148us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.861701 (+ 248us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:37.999766 (+138065us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.000871 (+ 1105us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.007714 (+ 6843us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53490: BlockingRecv error: recv got EOF from 127.0.0.1:53490 (error 108) 00:27:38.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":231} 00:27:38.010 [DEBUG - New I/O worker #1352] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.012 [INFO - New I/O worker #1381] (Connection.java:435) [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.012 [DEBUG - pool-3-thread-6] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.012 [DEBUG - New I/O worker #1381] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.013 [INFO - New I/O worker #1381] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.013 [DEBUG - pool-3-thread-12] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.018 [DEBUG - New I/O worker #1396] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.018 [DEBUG - New I/O worker #1401] (Negotiator.java:764) Authenticated connection [id: 0xa7440b64, /127.0.0.1:40120 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.019 [DEBUG - New I/O worker #1563] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.019 [DEBUG - New I/O worker #1489] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.020 [DEBUG - New I/O worker #1577] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.022 [DEBUG - New I/O worker #1379] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.025 [DEBUG - New I/O worker #1491] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.027 [DEBUG - New I/O worker #1330] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.032 [DEBUG - New I/O worker #1384] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.040 [DEBUG - New I/O worker #1488] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.053 [DEBUG - New I/O worker #1489] (Negotiator.java:764) Authenticated connection [id: 0xee70b620, /127.0.0.1:53520 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.058 [DEBUG - New I/O worker #1563] (Negotiator.java:764) Authenticated connection [id: 0x3e84a98f, /127.0.0.1:40150 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.058 [DEBUG - New I/O worker #1330] (Negotiator.java:764) Authenticated connection [id: 0xec32daa7, /127.0.0.1:38180 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.063 [DEBUG - New I/O worker #1488] (Negotiator.java:764) Authenticated connection [id: 0xd244a29e, /127.0.0.1:40156 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.063 [DEBUG - New I/O worker #1491] (Negotiator.java:764) Authenticated connection [id: 0x1167c6b9, /127.0.0.1:53514 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.064 [DEBUG - New I/O worker #1330] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.064 [DEBUG - New I/O worker #1330] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=107), Traces: [0ms] refreshing cache from master, [66ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [66ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [66ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [108ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.065 [DEBUG - New I/O worker #1379] (Negotiator.java:764) Authenticated connection [id: 0x3da1d918, /127.0.0.1:38168 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.066 [DEBUG - New I/O worker #1396] (Negotiator.java:764) Authenticated connection [id: 0xfc49c439, /127.0.0.1:38172 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.068 [DEBUG - New I/O worker #1577] (Negotiator.java:764) Authenticated connection [id: 0xec309c40, /127.0.0.1:40146 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.069 [DEBUG - New I/O worker #1396] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.069 [DEBUG - New I/O worker #1396] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=218), Traces: [0ms] refreshing cache from master, [161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [162ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [162ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [212ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [217ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.072 [DEBUG - New I/O worker #1379] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.072 [DEBUG - New I/O worker #1379] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=218), Traces: [0ms] refreshing cache from master, [158ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [158ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [159ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [211ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [216ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [216ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.072 [DEBUG - New I/O worker #1379] (Deferred.java:1330) callback=retry RPC@1977589726 returned Deferred@1910511037(state=PENDING, result=null, callback=(continuation of Deferred@887264741 after retry RPC@1977589726), errback=(continuation of Deferred@887264741 after retry RPC@1977589726)), so the following Deferred is getting paused: Deferred@887264741(state=PAUSED, result=Deferred@1910511037, callback=org.apache.kudu.client.AsyncKuduClient$6@c346ba2 -> wakeup thread pool-3-thread-62, errback=passthrough -> wakeup thread pool-3-thread-62) 00:27:38.073 [DEBUG - New I/O worker #1330] (Deferred.java:1330) callback=retry RPC@977233871 returned Deferred@1040249936(state=PENDING, result=null, callback=(continuation of Deferred@2063756788 after retry RPC@977233871), errback=(continuation of Deferred@2063756788 after retry RPC@977233871)), so the following Deferred is getting paused: Deferred@2063756788(state=PAUSED, result=Deferred@1040249936, callback=org.apache.kudu.client.AsyncKuduClient$6@bf3ec2b -> wakeup thread pool-3-thread-30, errback=passthrough -> wakeup thread pool-3-thread-30) 00:27:38.073 [DEBUG - New I/O worker #1330] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.072 [DEBUG - New I/O worker #1384] (Negotiator.java:764) Authenticated connection [id: 0x2e287029, /127.0.0.1:53526 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.075 [DEBUG - New I/O worker #1391] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.076 [DEBUG - New I/O worker #1379] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.076 [DEBUG - pool-3-thread-45] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.079 [DEBUG - New I/O worker #1330] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.079 [DEBUG - New I/O worker #1330] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.079 [DEBUG - New I/O worker #1330] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.079 [DEBUG - New I/O worker #1330] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.080 [DEBUG - New I/O worker #1330] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.080 [DEBUG - New I/O worker #1481] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.080 [DEBUG - pool-3-thread-30] (Deferred.java:1330) callback=disconnect callback@544310733 returned Deferred@62138949(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@106404323 after disconnect callback@544310733), errback=passthrough -> (continuation of Deferred@106404323 after disconnect callback@544310733)), so the following Deferred is getting paused: Deferred@106404323(state=PAUSED, result=Deferred@62138949, callback=, errback=) 00:27:38.080 [DEBUG - New I/O worker #1533] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.081 [DEBUG - New I/O worker #1384] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.082 [DEBUG - New I/O worker #1488] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.083 [DEBUG - New I/O worker #1330] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.084 [DEBUG - New I/O worker #1367] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.087 [DEBUG - New I/O worker #1379] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.088 [DEBUG - New I/O worker #1379] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.088 [DEBUG - New I/O worker #1379] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.089 [DEBUG - New I/O worker #1368] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.092 [DEBUG - pool-3-thread-25] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.092 [DEBUG - New I/O worker #1379] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.093 [DEBUG - New I/O worker #1379] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.093 [DEBUG - New I/O worker #1397] (Negotiator.java:764) Authenticated connection [id: 0x6f0d9cff, /127.0.0.1:38162 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.094 [DEBUG - pool-3-thread-62] (Deferred.java:1330) callback=disconnect callback@914745387 returned Deferred@645016918(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1512113263 after disconnect callback@914745387), errback=passthrough -> (continuation of Deferred@1512113263 after disconnect callback@914745387)), so the following Deferred is getting paused: Deferred@1512113263(state=PAUSED, result=Deferred@645016918, callback=, errback=) 00:27:38.094 [DEBUG - New I/O worker #1577] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.095 [DEBUG - New I/O worker #1491] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.096 [DEBUG - New I/O worker #1366] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.096 [DEBUG - New I/O worker #1379] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.098 [DEBUG - New I/O worker #1533] (Negotiator.java:764) Authenticated connection [id: 0xde15e8b7, /127.0.0.1:40162 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.103 [DEBUG - New I/O worker #1367] (Negotiator.java:764) Authenticated connection [id: 0xe3185f5e, /127.0.0.1:53538 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.111 [DEBUG - New I/O worker #1407] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.112 [DEBUG - pool-3-thread-20] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.119 [DEBUG - New I/O worker #1397] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.119 [DEBUG - New I/O worker #1397] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=349), Traces: [0ms] refreshing cache from master, [193ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [193ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [198ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [221ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [231ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [348ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.124 [DEBUG - New I/O worker #1481] (Negotiator.java:764) Authenticated connection [id: 0x7046b810, /127.0.0.1:53532 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.124 [DEBUG - New I/O worker #1407] (Negotiator.java:764) Authenticated connection [id: 0x617487f5, /127.0.0.1:40176 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.125 [DEBUG - New I/O worker #1397] (Deferred.java:1330) callback=retry RPC@1549957535 returned Deferred@1482553035(state=PENDING, result=null, callback=(continuation of Deferred@492794788 after retry RPC@1549957535), errback=(continuation of Deferred@492794788 after retry RPC@1549957535)), so the following Deferred is getting paused: Deferred@492794788(state=PAUSED, result=Deferred@1482553035, callback=org.apache.kudu.client.AsyncKuduClient$6@63e97df1 -> wakeup thread pool-3-thread-61, errback=passthrough -> wakeup thread pool-3-thread-61) 00:27:38.125 [DEBUG - New I/O worker #1397] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.126 [DEBUG - New I/O worker #1326] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.133 [DEBUG - pool-3-thread-69] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.134 [DEBUG - New I/O worker #1403] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.143 [DEBUG - New I/O worker #1325] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.143 [DEBUG - New I/O worker #1391] (Negotiator.java:764) Authenticated connection [id: 0x3a78f83c, /127.0.0.1:38186 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.146 [DEBUG - New I/O worker #1399] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.149 [DEBUG - New I/O worker #1329] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.150 [DEBUG - New I/O worker #1588] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.153 [DEBUG - New I/O worker #1377] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.163 [DEBUG - New I/O worker #1493] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.172 [DEBUG - New I/O worker #1366] (Negotiator.java:764) Authenticated connection [id: 0x6921f37f, /127.0.0.1:38192 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.175 [DEBUG - New I/O worker #1399] (Negotiator.java:764) Authenticated connection [id: 0xe5f1ecd2, /127.0.0.1:53552 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.176 [DEBUG - New I/O worker #1391] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.176 [DEBUG - New I/O worker #1391] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=215), Traces: [0ms] refreshing cache from master, [107ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [107ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [108ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [166ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [214ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.177 [DEBUG - New I/O worker #1391] (Deferred.java:1330) callback=retry RPC@108811290 returned Deferred@37994509(state=PENDING, result=null, callback=(continuation of Deferred@1195466273 after retry RPC@108811290), errback=(continuation of Deferred@1195466273 after retry RPC@108811290)), so the following Deferred is getting paused: Deferred@1195466273(state=PAUSED, result=Deferred@37994509, callback=org.apache.kudu.client.AsyncKuduClient$6@1274e5f7 -> wakeup thread pool-3-thread-80, errback=passthrough -> wakeup thread pool-3-thread-80) 00:27:38.179 [DEBUG - New I/O worker #1391] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.183 [DEBUG - New I/O worker #1397] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.205 [DEBUG - New I/O worker #1377] (Negotiator.java:764) Authenticated connection [id: 0x1cf1719d, /127.0.0.1:38212 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.205 [DEBUG - New I/O worker #1390] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.203 [DEBUG - New I/O worker #1403] (Negotiator.java:764) Authenticated connection [id: 0x054f417b, /127.0.0.1:40182 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.203 [DEBUG - New I/O worker #1405] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.202 [DEBUG - New I/O worker #1402] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.212 [DEBUG - New I/O worker #1329] (Negotiator.java:764) Authenticated connection [id: 0xada2a4ae, /127.0.0.1:38206 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.213 [DEBUG - New I/O worker #1377] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.214 [DEBUG - New I/O worker #1377] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=362), Traces: [0ms] refreshing cache from master, [292ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [292ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [293ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [362ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.214 [DEBUG - New I/O worker #1377] (Deferred.java:1330) callback=retry RPC@2021149913 returned Deferred@2085078798(state=PENDING, result=null, callback=(continuation of Deferred@960841442 after retry RPC@2021149913), errback=(continuation of Deferred@960841442 after retry RPC@2021149913)), so the following Deferred is getting paused: Deferred@960841442(state=PAUSED, result=Deferred@2085078798, callback=org.apache.kudu.client.AsyncKuduClient$6@3292c1e8 -> wakeup thread pool-3-thread-14, errback=passthrough -> wakeup thread pool-3-thread-14) 00:27:38.201 [DEBUG - New I/O worker #1588] (Negotiator.java:764) Authenticated connection [id: 0x6ee83936, /127.0.0.1:40188 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.193 [DEBUG - New I/O worker #1343] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.191 [DEBUG - New I/O worker #1325] (Negotiator.java:764) Authenticated connection [id: 0x1b16e4b4, /127.0.0.1:38200 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.189 [DEBUG - New I/O worker #1366] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.183 [DEBUG - New I/O worker #1326] (Negotiator.java:764) Authenticated connection [id: 0xd022c842, /127.0.0.1:53546 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.217 [DEBUG - New I/O worker #1377] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.217 [DEBUG - New I/O worker #1334] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.218 [DEBUG - New I/O worker #1397] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.218 [DEBUG - New I/O worker #1397] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.218 [DEBUG - New I/O worker #1397] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.183 [DEBUG - New I/O worker #1391] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.221 [DEBUG - New I/O worker #1391] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.221 [DEBUG - New I/O worker #1391] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.221 [DEBUG - New I/O worker #1391] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.222 [DEBUG - New I/O worker #1391] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.222 [DEBUG - New I/O worker #1377] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.222 [DEBUG - New I/O worker #1481] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.223 [DEBUG - New I/O worker #1396] (Deferred.java:1330) callback=retry RPC@200058868 returned Deferred@265499217(state=PENDING, result=null, callback=(continuation of Deferred@1255255503 after retry RPC@200058868), errback=(continuation of Deferred@1255255503 after retry RPC@200058868)), so the following Deferred is getting paused: Deferred@1255255503(state=PAUSED, result=Deferred@265499217, callback=org.apache.kudu.client.AsyncKuduClient$6@156c07e5 -> wakeup thread pool-3-thread-64, errback=passthrough -> wakeup thread pool-3-thread-64) 00:27:38.223 [DEBUG - New I/O worker #1533] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.225 [DEBUG - New I/O worker #1368] (Negotiator.java:764) Authenticated connection [id: 0x7b332dcd, /127.0.0.1:40168 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.221 [DEBUG - New I/O worker #1493] (Negotiator.java:764) Authenticated connection [id: 0x44da8e44, /127.0.0.1:53558 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.219 [DEBUG - New I/O worker #1325] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.218 [DEBUG - New I/O worker #1366] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=361), Traces: [0ms] refreshing cache from master, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [253ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [333ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.229 [DEBUG - New I/O worker #1366] (Deferred.java:1330) callback=retry RPC@742401859 returned Deferred@679454227(state=PENDING, result=null, callback=(continuation of Deferred@1836920184 after retry RPC@742401859), errback=(continuation of Deferred@1836920184 after retry RPC@742401859)), so the following Deferred is getting paused: Deferred@1836920184(state=PAUSED, result=Deferred@679454227, callback=org.apache.kudu.client.AsyncKuduClient$6@710d1961 -> wakeup thread pool-3-thread-65, errback=passthrough -> wakeup thread pool-3-thread-65) 00:27:38.218 [DEBUG - New I/O worker #1329] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.229 [DEBUG - New I/O worker #1329] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=383), Traces: [0ms] refreshing cache from master, [283ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [283ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [283ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [333ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [368ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [373ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.230 [DEBUG - New I/O worker #1329] (Deferred.java:1330) callback=retry RPC@702485666 returned Deferred@769688897(state=PENDING, result=null, callback=(continuation of Deferred@1759649433 after retry RPC@702485666), errback=(continuation of Deferred@1759649433 after retry RPC@702485666)), so the following Deferred is getting paused: Deferred@1759649433(state=PAUSED, result=Deferred@769688897, callback=org.apache.kudu.client.AsyncKuduClient$6@86b0cd2 -> wakeup thread pool-3-thread-21, errback=passthrough -> wakeup thread pool-3-thread-21) 00:27:38.229 [DEBUG - New I/O worker #1325] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=266), Traces: [0ms] refreshing cache from master, [145ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [146ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [146ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [165ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [257ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [257ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:38.225 [DEBUG - New I/O worker #1396] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.230 [DEBUG - New I/O worker #1325] (Deferred.java:1330) callback=retry RPC@2137981413 returned Deferred@2068880080(state=PENDING, result=null, callback=(continuation of Deferred@1045564382 after retry RPC@2137981413), errback=(continuation of Deferred@1045564382 after retry RPC@2137981413)), so the following Deferred is getting paused: Deferred@1045564382(state=PAUSED, result=Deferred@2068880080, callback=org.apache.kudu.client.AsyncKuduClient$6@841c845 -> wakeup thread pool-3-thread-2, errback=passthrough -> wakeup thread pool-3-thread-2) 00:27:38.231 [DEBUG - New I/O worker #1366] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.234 [DEBUG - New I/O worker #1405] (Negotiator.java:764) Authenticated connection [id: 0x07bf43cf, /127.0.0.1:40194 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.240 [DEBUG - New I/O worker #1396] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.240 [DEBUG - New I/O worker #1396] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.240 [DEBUG - New I/O worker #1396] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.240 [DEBUG - New I/O worker #1396] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.240 [DEBUG - New I/O worker #1396] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.241 [DEBUG - New I/O worker #1329] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.241 [DEBUG - pool-3-thread-64] (Deferred.java:1330) callback=disconnect callback@1007877098 returned Deferred@1000525424(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@131098703 after disconnect callback@1007877098), errback=passthrough -> (continuation of Deferred@131098703 after disconnect callback@1007877098)), so the following Deferred is getting paused: Deferred@131098703(state=PAUSED, result=Deferred@1000525424, callback=, errback=) 00:27:38.242 [DEBUG - New I/O worker #1489] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.242 [DEBUG - New I/O worker #1563] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.243 [DEBUG - New I/O worker #1396] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.244 [DEBUG - New I/O worker #1366] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.244 [DEBUG - New I/O worker #1366] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.244 [DEBUG - New I/O worker #1366] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.244 [DEBUG - New I/O worker #1366] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.245 [DEBUG - New I/O worker #1366] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.245 [DEBUG - pool-3-thread-65] (Deferred.java:1330) callback=disconnect callback@1213753077 returned Deferred@701431360(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1913222597 after disconnect callback@1213753077), errback=passthrough -> (continuation of Deferred@1913222597 after disconnect callback@1213753077)), so the following Deferred is getting paused: Deferred@1913222597(state=PAUSED, result=Deferred@701431360, callback=, errback=) 00:27:38.246 [DEBUG - New I/O worker #1367] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.246 [DEBUG - New I/O worker #1366] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.247 [DEBUG - New I/O worker #1368] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.223 [DEBUG - New I/O worker #1397] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.222 [DEBUG - New I/O worker #1391] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.248 [DEBUG - New I/O worker #1329] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.248 [DEBUG - New I/O worker #1329] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.249 [DEBUG - New I/O worker #1329] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.249 [DEBUG - New I/O worker #1475] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.250 [DEBUG - New I/O worker #1354] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.253 [DEBUG - New I/O worker #1355] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.258 [DEBUG - New I/O worker #1343] (Negotiator.java:764) Authenticated connection [id: 0x4869522d, /127.0.0.1:38218 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.261 [DEBUG - New I/O worker #1343] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.262 [DEBUG - New I/O worker #1343] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=411), Traces: [0ms] refreshing cache from master, [336ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [336ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [336ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [394ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [411ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.263 [DEBUG - New I/O worker #1397] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.222 [DEBUG - New I/O worker #1377] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.264 [DEBUG - New I/O worker #1377] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.264 [DEBUG - New I/O worker #1377] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.264 [DEBUG - New I/O worker #1377] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.264 [DEBUG - New I/O worker #1343] (Deferred.java:1330) callback=retry RPC@1425036483 returned Deferred@1355799173(state=PENDING, result=null, callback=(continuation of Deferred@365780728 after retry RPC@1425036483), errback=(continuation of Deferred@365780728 after retry RPC@1425036483)), so the following Deferred is getting paused: Deferred@365780728(state=PAUSED, result=Deferred@1355799173, callback=org.apache.kudu.client.AsyncKuduClient$6@7212fdad -> wakeup thread Time-limited test, errback=passthrough -> wakeup thread Time-limited test) 00:27:38.222 [DEBUG - pool-3-thread-80] (Deferred.java:1330) callback=disconnect callback@1740422539 returned Deferred@1284372236(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1368672296 after disconnect callback@1740422539), errback=passthrough -> (continuation of Deferred@1368672296 after disconnect callback@1740422539)), so the following Deferred is getting paused: Deferred@1368672296(state=PAUSED, result=Deferred@1284372236, callback=, errback=) 00:27:38.249 [DEBUG - New I/O worker #1329] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.249 [DEBUG - New I/O worker #1474] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.265 [DEBUG - New I/O worker #1588] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.265 [DEBUG - New I/O worker #1329] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.249 [DEBUG - pool-3-thread-61] (Deferred.java:1330) callback=disconnect callback@277295164 returned Deferred@804318261(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@244075504 after disconnect callback@277295164), errback=passthrough -> (continuation of Deferred@244075504 after disconnect callback@277295164)), so the following Deferred is getting paused: Deferred@244075504(state=PAUSED, result=Deferred@804318261, callback=, errback=) 00:27:38.266 [DEBUG - New I/O worker #1399] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.266 [DEBUG - New I/O worker #1403] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.238 [DEBUG - New I/O worker #1325] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.266 [DEBUG - New I/O worker #1377] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.266 [DEBUG - pool-3-thread-21] (Deferred.java:1330) callback=disconnect callback@389981523 returned Deferred@1523894205(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@663800405 after disconnect callback@389981523), errback=passthrough -> (continuation of Deferred@663800405 after disconnect callback@389981523)), so the following Deferred is getting paused: Deferred@663800405(state=PAUSED, result=Deferred@1523894205, callback=, errback=) 00:27:38.265 [DEBUG - pool-3-thread-14] (Deferred.java:1330) callback=disconnect callback@1069120275 returned Deferred@2056818610(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@157349848 after disconnect callback@1069120275), errback=passthrough -> (continuation of Deferred@157349848 after disconnect callback@1069120275)), so the following Deferred is getting paused: Deferred@157349848(state=PAUSED, result=Deferred@2056818610, callback=, errback=) 00:27:38.269 [DEBUG - New I/O worker #1329] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.272 [DEBUG - New I/O worker #1325] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.273 [DEBUG - New I/O worker #1325] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.273 [DEBUG - New I/O worker #1325] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.273 [DEBUG - New I/O worker #1325] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.273 [DEBUG - New I/O worker #1325] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.274 [DEBUG - pool-3-thread-2] (Deferred.java:1330) callback=disconnect callback@1930068863 returned Deferred@621174202(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1996193454 after disconnect callback@1930068863), errback=passthrough -> (continuation of Deferred@1996193454 after disconnect callback@1930068863)), so the following Deferred is getting paused: Deferred@1996193454(state=PAUSED, result=Deferred@621174202, callback=, errback=) 00:27:38.274 [DEBUG - New I/O worker #1407] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.275 [DEBUG - New I/O worker #1325] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.272 [DEBUG - New I/O worker #1402] (Negotiator.java:764) Authenticated connection [id: 0x976d19d1, /127.0.0.1:53564 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.275 [DEBUG - New I/O worker #1493] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.274 [DEBUG - New I/O worker #1326] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.277 [DEBUG - New I/O worker #1379] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.278 [DEBUG - New I/O worker #1330] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.301 [DEBUG - pool-3-thread-10] (Deferred.java:1330) callback=disconnect callback@1911489291 returned Deferred@2021037498(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@183031992 after disconnect callback@1911489291), errback=passthrough -> (continuation of Deferred@183031992 after disconnect callback@1911489291)), so the following Deferred is getting paused: Deferred@183031992(state=PAUSED, result=Deferred@2021037498, callback=, errback=) 00:27:38.301 [DEBUG - New I/O worker #1401] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.304 [DEBUG - New I/O worker #1347] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.316 [DEBUG - pool-3-thread-46] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.317 [DEBUG - New I/O worker #1395] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.320 [DEBUG - New I/O worker #1334] (Negotiator.java:764) Authenticated connection [id: 0xdd709536, /127.0.0.1:38198 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.323 [DEBUG - New I/O worker #1334] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.323 [DEBUG - New I/O worker #1334] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=475), Traces: [0ms] refreshing cache from master, [251ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [371ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [371ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [475ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.324 [DEBUG - New I/O worker #1401] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.324 [DEBUG - New I/O worker #1334] (Deferred.java:1330) callback=retry RPC@660682410 returned Deferred@593405598(state=PENDING, result=null, callback=(continuation of Deferred@1717320849 after retry RPC@660682410), errback=(continuation of Deferred@1717320849 after retry RPC@660682410)), so the following Deferred is getting paused: Deferred@1717320849(state=PAUSED, result=Deferred@593405598, callback=org.apache.kudu.client.AsyncKuduClient$6@798a5409 -> wakeup thread pool-3-thread-37, errback=passthrough -> wakeup thread pool-3-thread-37) 00:27:38.325 [DEBUG - pool-3-thread-53] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.326 [DEBUG - New I/O worker #1343] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.326 [DEBUG - New I/O worker #1334] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.327 [DEBUG - pool-3-thread-5] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.330 [DEBUG - New I/O worker #1334] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.330 [DEBUG - New I/O worker #1334] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.330 [DEBUG - New I/O worker #1334] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.331 [DEBUG - New I/O worker #1334] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.331 [DEBUG - New I/O worker #1334] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.336 [DEBUG - pool-3-thread-13] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.336 [DEBUG - New I/O worker #1493] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.337 [DEBUG - New I/O worker #1325] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.338 [DEBUG - New I/O worker #1329] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.338 [DEBUG - New I/O worker #1397] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.339 [DEBUG - New I/O worker #1368] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.339 [DEBUG - New I/O worker #1396] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.340 [DEBUG - New I/O worker #1533] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:38.333106 25053 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:38.350 [DEBUG - New I/O worker #1334] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.219887 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53568 00:27:38.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.220350 (+ 463us) server_negotiation.cc:184] Beginning negotiation 00:27:38.350 [DEBUG - pool-3-thread-37] (Deferred.java:1330) callback=disconnect callback@607562064 returned Deferred@1894944338(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1710737185 after disconnect callback@607562064), errback=passthrough -> (continuation of Deferred@1710737185 after disconnect callback@607562064)), so the following Deferred is getting paused: Deferred@1710737185(state=PAUSED, result=Deferred@1894944338, callback=, errback=) 00:27:38.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.220375 (+ 25us) server_negotiation.cc:373] Waiting for connection header 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.249803 (+ 29428us) server_negotiation.cc:381] Connection header received 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.249933 (+ 130us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.249959 (+ 26us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.250092 (+ 133us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.250266 (+ 174us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.251611 (+ 1345us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.252773 (+ 1162us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.332482 (+ 79709us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53568: BlockingRecv error: recv got EOF from 127.0.0.1:53568 (error 108) 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":148} 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:38.334777 24989 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.219828 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40198 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.220318 (+ 490us) server_negotiation.cc:184] Beginning negotiation 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.220339 (+ 21us) server_negotiation.cc:373] Waiting for connection header 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.252244 (+ 31905us) server_negotiation.cc:381] Connection header received 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.252371 (+ 127us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.252396 (+ 25us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:38.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.252676 (+ 280us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:38.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.253978 (+ 1302us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:38.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.281582 (+ 27604us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:38.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.285462 (+ 3880us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:38.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.334155 (+ 48693us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40198: BlockingRecv error: recv got EOF from 127.0.0.1:40198 (error 108) 00:27:38.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":177} 00:27:38.351 [DEBUG - New I/O worker #1343] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.351 [DEBUG - New I/O worker #1334] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.352 [DEBUG - New I/O worker #1343] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.352 [DEBUG - New I/O worker #1343] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.353 [DEBUG - New I/O worker #1343] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.356 [INFO - New I/O worker #1354] (Connection.java:435) [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.356 [DEBUG - New I/O worker #1354] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.356 [INFO - New I/O worker #1354] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.368 [INFO - New I/O worker #1355] (Connection.java:435) [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:38.368 [DEBUG - New I/O worker #1355] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:38.368 [INFO - New I/O worker #1355] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:38.379 [DEBUG - New I/O worker #1328] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.381 [DEBUG - New I/O worker #1570] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.382 [DEBUG - New I/O worker #1392] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.383 [DEBUG - New I/O worker #1494] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.396 [DEBUG - New I/O worker #1369] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.397 [DEBUG - New I/O worker #1343] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299955, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299955, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299955, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299955, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.398 [DEBUG - Time-limited test] (Deferred.java:1330) callback=disconnect callback@49622167 returned Deferred@863573086(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1338890489 after disconnect callback@49622167), errback=passthrough -> (continuation of Deferred@1338890489 after disconnect callback@49622167)), so the following Deferred is getting paused: Deferred@1338890489(state=PAUSED, result=Deferred@863573086, callback=, errback=) 00:27:38.399 [DEBUG - New I/O worker #1402] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.399 [DEBUG - New I/O worker #1405] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.400 [DEBUG - New I/O worker #1343] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.401 [DEBUG - pool-3-thread-40] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.403 [DEBUG - New I/O worker #1328] (Negotiator.java:764) Authenticated connection [id: 0xe2dc767b, /127.0.0.1:38228 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.406 [DEBUG - New I/O worker #1328] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.406 [DEBUG - New I/O worker #1328] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=408), Traces: [0ms] refreshing cache from master, [305ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [305ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [305ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [408ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.407 [DEBUG - New I/O worker #1328] (Deferred.java:1330) callback=retry RPC@915037206 returned Deferred@850775583(state=PENDING, result=null, callback=(continuation of Deferred@2008503853 after retry RPC@915037206), errback=(continuation of Deferred@2008503853 after retry RPC@915037206)), so the following Deferred is getting paused: Deferred@2008503853(state=PAUSED, result=Deferred@850775583, callback=org.apache.kudu.client.AsyncKuduClient$6@2284f369 -> wakeup thread pool-3-thread-4, errback=passthrough -> wakeup thread pool-3-thread-4) 00:27:38.409 [DEBUG - New I/O worker #1328] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.413 [DEBUG - New I/O worker #1328] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.413 [DEBUG - New I/O worker #1328] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.413 [DEBUG - New I/O worker #1328] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.413 [DEBUG - New I/O worker #1328] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.413 [DEBUG - New I/O worker #1328] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.414 [DEBUG - New I/O worker #1328] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:38.415940 25003 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.304034 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53574 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.304481 (+ 447us) server_negotiation.cc:184] Beginning negotiation 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.304506 (+ 25us) server_negotiation.cc:373] Waiting for connection header 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.304799 (+ 293us) server_negotiation.cc:381] Connection header received 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.391681 (+ 86882us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.391712 (+ 31us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.391912 (+ 200us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.392075 (+ 163us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.402082 (+ 10007us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.403177 (+ 1095us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.414966 (+ 11789us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53574: BlockingRecv error: recv got EOF from 127.0.0.1:53574 (error 108) 00:27:38.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":129} 00:27:38.420 [DEBUG - New I/O worker #1494] (Negotiator.java:764) Authenticated connection [id: 0xb2432237, /127.0.0.1:53590 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.434 [DEBUG - New I/O worker #1539] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.434 [DEBUG - New I/O worker #1349] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.435 [DEBUG - New I/O worker #1380] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.436 [DEBUG - pool-3-thread-26] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.439 [DEBUG - pool-3-thread-56] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.439 [DEBUG - New I/O worker #1570] (Negotiator.java:764) Authenticated connection [id: 0x17a19fbd, /127.0.0.1:40214 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.441 [DEBUG - New I/O worker #1375] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.448 [DEBUG - pool-3-thread-74] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.458 [DEBUG - New I/O worker #1380] (Negotiator.java:764) Authenticated connection [id: 0xf9649941, /127.0.0.1:38238 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.462 [DEBUG - New I/O worker #1539] (Negotiator.java:764) Authenticated connection [id: 0xa0d18c52, /127.0.0.1:40244 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.462 [DEBUG - New I/O worker #1380] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.463 [DEBUG - New I/O worker #1380] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=386), Traces: [0ms] refreshing cache from master, [259ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [260ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [260ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [369ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [386ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.463 [DEBUG - New I/O worker #1380] (Deferred.java:1330) callback=retry RPC@262217358 returned Deferred@194939810(state=PENDING, result=null, callback=(continuation of Deferred@1318851765 after retry RPC@262217358), errback=(continuation of Deferred@1318851765 after retry RPC@262217358)), so the following Deferred is getting paused: Deferred@1318851765(state=PAUSED, result=Deferred@194939810, callback=org.apache.kudu.client.AsyncKuduClient$6@78ca1dc4 -> wakeup thread pool-3-thread-45, errback=passthrough -> wakeup thread pool-3-thread-45) 00:27:38.464 [DEBUG - New I/O worker #1598] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.465 [DEBUG - New I/O worker #1490] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.467 [DEBUG - New I/O worker #1380] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.467 [DEBUG - New I/O worker #1387] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.468 [DEBUG - New I/O worker #1333] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.469 [DEBUG - New I/O worker #1370] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.470 [DEBUG - New I/O worker #1363] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.471 [DEBUG - New I/O worker #1380] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.471 [DEBUG - pool-3-thread-79] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.471 [DEBUG - New I/O worker #1380] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.471 [DEBUG - New I/O worker #1380] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.473 [DEBUG - New I/O worker #1332] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.473 [DEBUG - New I/O worker #1335] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.473 [DEBUG - New I/O worker #1338] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.473 [DEBUG - New I/O worker #1487] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.476 [DEBUG - New I/O worker #1380] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.476 [DEBUG - New I/O worker #1380] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.477 [DEBUG - New I/O worker #1380] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:38.478451 25051 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:38.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.338917 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53584 00:27:38.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.339691 (+ 774us) server_negotiation.cc:184] Beginning negotiation 00:27:38.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.339717 (+ 26us) server_negotiation.cc:373] Waiting for connection header 00:27:38.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.339757 (+ 40us) server_negotiation.cc:381] Connection header received 00:27:38.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.381027 (+ 41270us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:38.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.381055 (+ 28us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:38.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.381178 (+ 123us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:38.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.381340 (+ 162us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:38.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.469038 (+ 87698us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:38.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.470249 (+ 1211us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:38.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.477574 (+ 7325us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53584: BlockingRecv error: recv got EOF from 127.0.0.1:53584 (error 108) 00:27:38.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":355} 00:27:38.479 [DEBUG - New I/O worker #1351] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.481 [DEBUG - New I/O worker #1336] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.484 [DEBUG - New I/O worker #1364] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.489 [DEBUG - New I/O worker #1482] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.489 [DEBUG - New I/O worker #1620] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.491 [DEBUG - New I/O worker #1376] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.494 [DEBUG - New I/O worker #1363] (Negotiator.java:764) Authenticated connection [id: 0x80425e0d, /127.0.0.1:53640 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.496 [DEBUG - New I/O worker #1436] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.497 [DEBUG - New I/O worker #1445] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.498 [DEBUG - New I/O worker #1375] (Negotiator.java:764) Authenticated connection [id: 0xf7245d2b, /127.0.0.1:38234 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.498 [DEBUG - New I/O worker #1341] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.499 [DEBUG - New I/O worker #1344] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.499 [DEBUG - New I/O worker #1385] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.499 [DEBUG - New I/O worker #1457] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.500 [DEBUG - New I/O worker #1492] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.501 [DEBUG - New I/O worker #1375] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.501 [DEBUG - New I/O worker #1374] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.501 [DEBUG - New I/O worker #1386] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.507 [DEBUG - New I/O worker #1524] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.510 [DEBUG - New I/O worker #1398] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.527 [DEBUG - New I/O worker #1335] (Negotiator.java:764) Authenticated connection [id: 0xe507ccf5, /127.0.0.1:38236 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.527 [DEBUG - New I/O worker #1343] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.534 [DEBUG - New I/O worker #1374] (Negotiator.java:764) Authenticated connection [id: 0xb3b2fbac, /127.0.0.1:38268 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.534 [DEBUG - New I/O worker #1335] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.534 [DEBUG - New I/O worker #1335] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=522), Traces: [0ms] refreshing cache from master, [323ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [391ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [391ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [522ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.539 [DEBUG - New I/O worker #1340] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.540 [DEBUG - New I/O worker #1335] (Deferred.java:1330) callback=retry RPC@1442350880 returned Deferred@1372002631(state=PENDING, result=null, callback=(continuation of Deferred@348491035 after retry RPC@1442350880), errback=(continuation of Deferred@348491035 after retry RPC@1442350880)), so the following Deferred is getting paused: Deferred@348491035(state=PAUSED, result=Deferred@1372002631, callback=org.apache.kudu.client.AsyncKuduClient$6@34cbfb7e -> wakeup thread pool-3-thread-6, errback=passthrough -> wakeup thread pool-3-thread-6) 00:27:38.540 [DEBUG - New I/O worker #1413] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.541 [DEBUG - New I/O worker #1335] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.541 [INFO - New I/O worker #1490] (Connection.java:435) [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.541 [DEBUG - New I/O worker #1490] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.542 [INFO - New I/O worker #1490] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.547 [DEBUG - New I/O worker #1335] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.614 [DEBUG - New I/O worker #1374] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.615 [DEBUG - New I/O worker #1374] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=522), Traces: [0ms] refreshing cache from master, [284ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [285ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [285ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [375ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [522ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.610 [DEBUG - New I/O worker #1436] (Negotiator.java:764) Authenticated connection [id: 0xf1bf303b, /127.0.0.1:53628 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.553 [DEBUG - New I/O worker #1370] (Negotiator.java:764) Authenticated connection [id: 0xd65a7668, /127.0.0.1:40204 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.606 [DEBUG - New I/O worker #1338] (Negotiator.java:764) Authenticated connection [id: 0x1cf8bf98, /127.0.0.1:38274 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.602 [DEBUG - New I/O worker #1487] (Negotiator.java:764) Authenticated connection [id: 0x0dfcfef8, /127.0.0.1:40264 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.600 [DEBUG - New I/O worker #1341] (Negotiator.java:764) Authenticated connection [id: 0x3da7563d, /127.0.0.1:38256 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.631 [DEBUG - New I/O worker #1341] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.632 [DEBUG - New I/O worker #1341] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=307), Traces: [0ms] refreshing cache from master, [17ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [17ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [17ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [305ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.633 [DEBUG - New I/O worker #1341] (Deferred.java:1330) callback=retry RPC@485256706 returned Deferred@416540999(state=PENDING, result=null, callback=(continuation of Deferred@1574003769 after retry RPC@485256706), errback=(continuation of Deferred@1574003769 after retry RPC@485256706)), so the following Deferred is getting paused: Deferred@1574003769(state=PAUSED, result=Deferred@416540999, callback=org.apache.kudu.client.AsyncKuduClient$6@13256f8e -> wakeup thread pool-3-thread-53, errback=passthrough -> wakeup thread pool-3-thread-53) 00:27:38.599 [DEBUG - New I/O worker #1351] (Negotiator.java:764) Authenticated connection [id: 0xf2ed49e8, /127.0.0.1:53620 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.589 [DEBUG - New I/O worker #1336] (Negotiator.java:764) Authenticated connection [id: 0xd7cc92ae, /127.0.0.1:38294 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.581 [DEBUG - New I/O worker #1524] (Negotiator.java:764) Authenticated connection [id: 0x886fd7f0, /127.0.0.1:40260 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.636 [DEBUG - New I/O worker #1336] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.637 [DEBUG - New I/O worker #1336] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=308), Traces: [0ms] refreshing cache from master, [109ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [110ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [110ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [182ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [309ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.637 [DEBUG - New I/O worker #1341] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.566 [DEBUG - New I/O worker #1387] (Negotiator.java:764) Authenticated connection [id: 0xc79202ca, /127.0.0.1:40226 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.566 [DEBUG - New I/O worker #1340] (Negotiator.java:764) Authenticated connection [id: 0xefdcb433, /127.0.0.1:38300 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.565 [DEBUG - New I/O worker #1598] (Negotiator.java:764) Authenticated connection [id: 0xf94f9e12, /127.0.0.1:40220 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.558 [DEBUG - New I/O worker #1414] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.554 [DEBUG - New I/O worker #1333] (Negotiator.java:764) Authenticated connection [id: 0x8a2901da, /127.0.0.1:38250 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.642 [DEBUG - New I/O worker #1335] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.642 [DEBUG - New I/O worker #1335] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.643 [DEBUG - New I/O worker #1335] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.643 [DEBUG - New I/O worker #1333] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.643 [DEBUG - New I/O worker #1333] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=307), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [307ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.644 [DEBUG - New I/O worker #1333] (Deferred.java:1330) callback=retry RPC@1270552647 returned Deferred@1334089192(state=PENDING, result=null, callback=(continuation of Deferred@176558716 after retry RPC@1270552647), errback=(continuation of Deferred@176558716 after retry RPC@1270552647)), so the following Deferred is getting paused: Deferred@176558716(state=PAUSED, result=Deferred@1334089192, callback=org.apache.kudu.client.AsyncKuduClient$6@282ca58a -> wakeup thread pool-3-thread-13, errback=passthrough -> wakeup thread pool-3-thread-13) 00:27:38.645 [DEBUG - New I/O worker #1341] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.645 [DEBUG - New I/O worker #1341] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.645 [DEBUG - New I/O worker #1341] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.646 [DEBUG - New I/O worker #1341] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.646 [DEBUG - New I/O worker #1341] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.646 [DEBUG - New I/O worker #1340] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:38.649551 25621 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.342716 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53602 00:27:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.344237 (+ 1521us) server_negotiation.cc:184] Beginning negotiation 00:27:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.344258 (+ 21us) server_negotiation.cc:373] Waiting for connection header 00:27:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.394448 (+ 50190us) server_negotiation.cc:381] Connection header received 00:27:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.433854 (+ 39406us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.433882 (+ 28us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.434001 (+ 119us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.434144 (+ 143us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.502503 (+ 68359us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.505603 (+ 3100us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.648493 (+142890us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53602: BlockingRecv error: recv got EOF from 127.0.0.1:53602 (error 108) 00:27:38.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":1257,"thread_start_us":1039,"threads_started":1} 00:27:38.651 [DEBUG - New I/O worker #1341] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.653 [DEBUG - New I/O worker #1333] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.637 [DEBUG - New I/O worker #1336] (Deferred.java:1330) callback=retry RPC@682201496 returned Deferred@748027989(state=PENDING, result=null, callback=(continuation of Deferred@1771346851 after retry RPC@682201496), errback=(continuation of Deferred@1771346851 after retry RPC@682201496)), so the following Deferred is getting paused: Deferred@1771346851(state=PAUSED, result=Deferred@748027989, callback=org.apache.kudu.client.AsyncKuduClient$6@3cf6f2f7 -> wakeup thread pool-3-thread-5, errback=passthrough -> wakeup thread pool-3-thread-5) 00:27:38.632 [DEBUG - New I/O worker #1338] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.655 [DEBUG - New I/O worker #1338] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=521), Traces: [0ms] refreshing cache from master, [243ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [243ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [243ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [498ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [504ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:38.657 [DEBUG - New I/O worker #1338] (Deferred.java:1330) callback=retry RPC@655630053 returned Deferred@590061279(state=PENDING, result=null, callback=(continuation of Deferred@1713968350 after retry RPC@655630053), errback=(continuation of Deferred@1713968350 after retry RPC@655630053)), so the following Deferred is getting paused: Deferred@1713968350(state=PAUSED, result=Deferred@590061279, callback=org.apache.kudu.client.AsyncKuduClient$6@5061eb39 -> wakeup thread pool-3-thread-69, errback=passthrough -> wakeup thread pool-3-thread-69) 00:27:38.654 [DEBUG - New I/O worker #1335] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299988, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.657 [DEBUG - New I/O worker #1333] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.658 [DEBUG - New I/O worker #1333] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.658 [DEBUG - New I/O worker #1333] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.658 [DEBUG - New I/O worker #1333] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.659 [DEBUG - New I/O worker #1333] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.659 [DEBUG - New I/O worker #1338] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.659 [DEBUG - pool-3-thread-13] (Deferred.java:1330) callback=disconnect callback@1835075742 returned Deferred@232189238(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1816721817 after disconnect callback@1835075742), errback=passthrough -> (continuation of Deferred@1816721817 after disconnect callback@1835075742)), so the following Deferred is getting paused: Deferred@1816721817(state=PAUSED, result=Deferred@232189238, callback=, errback=) 00:27:38.659 [DEBUG - New I/O worker #1333] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.660 [DEBUG - New I/O worker #1503] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.661 [DEBUG - New I/O worker #1335] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.662 [DEBUG - New I/O worker #1365] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.663 [DEBUG - New I/O worker #1337] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.662 [DEBUG - New I/O worker #1340] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=330), Traces: [0ms] refreshing cache from master, [198ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [199ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [199ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [330ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.665 [DEBUG - New I/O worker #1497] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.665 [DEBUG - New I/O worker #1340] (Deferred.java:1330) callback=retry RPC@2077523109 returned Deferred@2146166535(state=PENDING, result=null, callback=(continuation of Deferred@988378782 after retry RPC@2077523109), errback=(continuation of Deferred@988378782 after retry RPC@2077523109)), so the following Deferred is getting paused: Deferred@988378782(state=PAUSED, result=Deferred@2146166535, callback=org.apache.kudu.client.AsyncKuduClient$6@53702580 -> wakeup thread pool-3-thread-46, errback=passthrough -> wakeup thread pool-3-thread-46) 00:27:38.666 [DEBUG - New I/O worker #1502] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.667 [DEBUG - New I/O worker #1607] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.662 [DEBUG - New I/O worker #1336] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.810 [DEBUG - New I/O worker #1398] (Negotiator.java:764) Authenticated connection [id: 0x51f17fbf, /127.0.0.1:53608 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.807 [DEBUG - New I/O worker #1393] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.810 [DEBUG - New I/O worker #1503] (Negotiator.java:764) Authenticated connection [id: 0x18783fdd, /127.0.0.1:40282 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.805 [DEBUG - New I/O worker #1448] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.805 [DEBUG - New I/O worker #1457] (Negotiator.java:764) Authenticated connection [id: 0x633c98f0, /127.0.0.1:40238 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.805 [DEBUG - New I/O worker #1345] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.805 [DEBUG - New I/O worker #1350] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.805 [DEBUG - New I/O worker #1364] (Negotiator.java:764) Authenticated connection [id: 0xab3c7714, /127.0.0.1:40270 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.805 [DEBUG - New I/O worker #1383] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.804 [DEBUG - New I/O worker #1413] (Negotiator.java:764) Authenticated connection [id: 0xf24e31cb, /127.0.0.1:53646 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.804 [DEBUG - New I/O worker #1332] (Negotiator.java:764) Authenticated connection [id: 0x4a14771e, /127.0.0.1:38282 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.804 [DEBUG - New I/O worker #1414] (Negotiator.java:764) Authenticated connection [id: 0x650ee7e8, /127.0.0.1:40276 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.802 [DEBUG - New I/O worker #1620] (Negotiator.java:764) Authenticated connection [id: 0x8a4ff68c, /127.0.0.1:40250 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.802 [DEBUG - New I/O worker #1482] (Negotiator.java:764) Authenticated connection [id: 0x64b953d6, /127.0.0.1:53614 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.802 [DEBUG - New I/O worker #1374] (Deferred.java:1330) callback=retry RPC@1891756097 returned Deferred@1962832513(state=PENDING, result=null, callback=(continuation of Deferred@838656634 after retry RPC@1891756097), errback=(continuation of Deferred@838656634 after retry RPC@1891756097)), so the following Deferred is getting paused: Deferred@838656634(state=PAUSED, result=Deferred@1962832513, callback=org.apache.kudu.client.AsyncKuduClient$6@7f7bac25 -> wakeup thread pool-3-thread-25, errback=passthrough -> wakeup thread pool-3-thread-25) 00:27:38.802 [DEBUG - New I/O worker #1387] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.801 [INFO - New I/O worker #1369] (Connection.java:435) [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.819 [DEBUG - New I/O worker #1369] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.820 [INFO - New I/O worker #1369] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.801 [DEBUG - New I/O worker #1338] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.820 [DEBUG - New I/O worker #1338] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.821 [DEBUG - New I/O worker #1338] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.821 [DEBUG - New I/O worker #1338] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.821 [DEBUG - New I/O worker #1338] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.801 [DEBUG - New I/O worker #1392] (Negotiator.java:764) Authenticated connection [id: 0xd67b8fbb, /127.0.0.1:38244 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.801 [DEBUG - New I/O worker #1500] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.822 [DEBUG - pool-3-thread-69] (Deferred.java:1330) callback=disconnect callback@1306216702 returned Deferred@850159345(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@754939247 after disconnect callback@1306216702), errback=passthrough -> (continuation of Deferred@754939247 after disconnect callback@1306216702)), so the following Deferred is getting paused: Deferred@754939247(state=PAUSED, result=Deferred@850159345, callback=, errback=) 00:27:38.822 [DEBUG - New I/O worker #1351] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.823 [DEBUG - New I/O worker #1620] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.823 [INFO - New I/O worker #1620] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:38.799 [DEBUG - New I/O worker #1483] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.798 [DEBUG - New I/O worker #1376] (Negotiator.java:764) Authenticated connection [id: 0x473d5eb2, /127.0.0.1:53624 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.798 [DEBUG - New I/O worker #1445] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.797 [DEBUG - New I/O worker #1340] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.796 [DEBUG - New I/O worker #1492] (Negotiator.java:764) Authenticated connection [id: 0xf0395afc, /127.0.0.1:40256 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.796 [DEBUG - New I/O worker #1378] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:38.796 [DEBUG - New I/O worker #1344] (Negotiator.java:764) Authenticated connection [id: 0x69e04604, /127.0.0.1:38262 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.796 [DEBUG - New I/O worker #1385] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:38.795310 25053 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.339121 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53596 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.340784 (+ 1663us) server_negotiation.cc:184] Beginning negotiation 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.340810 (+ 26us) server_negotiation.cc:373] Waiting for connection header 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.340850 (+ 40us) server_negotiation.cc:381] Connection header received 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.402419 (+ 61569us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.402448 (+ 29us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.402588 (+ 140us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.402726 (+ 138us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.503027 (+100301us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.504171 (+ 1144us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.641308 (+137137us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.642694 (+ 1386us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.642898 (+ 204us) server_negotiation.cc:589] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.793805 (+150907us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53596: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:53596): Cannot send after transport endpoint shutdown (error 108) 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":1419} 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:38.800274 24989 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.342863 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40232 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.344738 (+ 1875us) server_negotiation.cc:184] Beginning negotiation 00:27:38.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.344764 (+ 26us) server_negotiation.cc:373] Waiting for connection header 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.435449 (+ 90685us) server_negotiation.cc:381] Connection header received 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.435572 (+ 123us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.435599 (+ 27us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.435720 (+ 121us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.435879 (+ 159us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.502797 (+ 66918us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.503908 (+ 1111us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.619003 (+115095us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.623556 (+ 4553us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.623779 (+ 223us) server_negotiation.cc:589] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.799045 (+175266us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40232: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:40232): Cannot send after transport endpoint shutdown (error 108) 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":3132,"server-negotiator.queue_time_us":1614} 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:38.808488 25003 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.404336 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53634 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.417165 (+ 12829us) server_negotiation.cc:184] Beginning negotiation 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.417210 (+ 45us) server_negotiation.cc:373] Waiting for connection header 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.431919 (+ 14709us) server_negotiation.cc:381] Connection header received 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.432059 (+ 140us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.432085 (+ 26us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:38.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.432225 (+ 140us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:38.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.432505 (+ 280us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:38.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.503552 (+ 71047us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:38.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.504727 (+ 1175us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:38.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.624665 (+119938us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:38.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.627135 (+ 2470us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:38.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.627376 (+ 241us) server_negotiation.cc:589] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 00:27:38.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:38.797851 (+170475us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53634: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:53634): Cannot send after transport endpoint shutdown (error 108) 00:27:38.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":1389,"server-negotiator.queue_time_us":12547,"thread_start_us":39313,"threads_started":1} 00:27:38.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:38.827539 22690 connection.cc:554] Connection torn down before Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:40250 (request call id 0) could send its response 00:27:38.795 [DEBUG - New I/O worker #1487] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.794 [DEBUG - New I/O worker #1346] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:38.792 [DEBUG - New I/O worker #1386] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:38.845 [INFO - New I/O worker #1386] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:38.792 [DEBUG - New I/O worker #1370] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.792 [DEBUG - pool-3-thread-6] (Deferred.java:1330) callback=disconnect callback@1475120734 returned Deferred@2099617353(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1259428313 after disconnect callback@1475120734), errback=passthrough -> (continuation of Deferred@1259428313 after disconnect callback@1475120734)), so the following Deferred is getting paused: Deferred@1259428313(state=PAUSED, result=Deferred@2099617353, callback=, errback=) 00:27:38.792 [DEBUG - pool-3-thread-4] (Deferred.java:1330) callback=disconnect callback@1171478112 returned Deferred@807645052(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1092830647 after disconnect callback@1171478112), errback=passthrough -> (continuation of Deferred@1092830647 after disconnect callback@1171478112)), so the following Deferred is getting paused: Deferred@1092830647(state=PAUSED, result=Deferred@807645052, callback=, errback=) 00:27:38.791 [INFO - New I/O worker #1349] (Connection.java:435) [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.779 [DEBUG - New I/O worker #1375] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=561), Traces: [0ms] refreshing cache from master, [362ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [362ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [435ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [558ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.846 [DEBUG - New I/O worker #1375] (Deferred.java:1330) callback=retry RPC@1538031191 returned Deferred@1603527162(state=PENDING, result=null, callback=(continuation of Deferred@445740140 after retry RPC@1538031191), errback=(continuation of Deferred@445740140 after retry RPC@1538031191)), so the following Deferred is getting paused: Deferred@445740140(state=PAUSED, result=Deferred@1603527162, callback=org.apache.kudu.client.AsyncKuduClient$6@13a7826e -> wakeup thread pool-3-thread-42, errback=passthrough -> wakeup thread pool-3-thread-42) 00:27:38.779 [DEBUG - pool-3-thread-53] (Deferred.java:1330) callback=disconnect callback@1794459389 returned Deferred@541396227(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1475886038 after disconnect callback@1794459389), errback=passthrough -> (continuation of Deferred@1475886038 after disconnect callback@1794459389)), so the following Deferred is getting paused: Deferred@1475886038(state=PAUSED, result=Deferred@541396227, callback=, errback=) 00:27:38.778 [DEBUG - New I/O worker #1570] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.778 [DEBUG - pool-3-thread-45] (Deferred.java:1330) callback=disconnect callback@1359369680 returned Deferred@1153799835(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2028084861 after disconnect callback@1359369680), errback=passthrough -> (continuation of Deferred@2028084861 after disconnect callback@1359369680)), so the following Deferred is getting paused: Deferred@2028084861(state=PAUSED, result=Deferred@1153799835, callback=, errback=) 00:27:38.866 [DEBUG - pool-3-thread-37] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.863 [DEBUG - New I/O worker #1502] (Negotiator.java:764) Authenticated connection [id: 0xe8e0173c, /127.0.0.1:40294 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.863 [DEBUG - New I/O worker #1393] (Negotiator.java:764) Authenticated connection [id: 0x98fd670a, /127.0.0.1:38330 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.870 [DEBUG - New I/O worker #1393] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.871 [DEBUG - New I/O worker #1393] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=434), Traces: [0ms] refreshing cache from master, [362ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [362ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [362ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [434ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.860 [DEBUG - New I/O worker #1497] (Negotiator.java:764) Authenticated connection [id: 0x02c5d6b4, /127.0.0.1:53658 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.856 [DEBUG - New I/O worker #1332] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.874 [DEBUG - New I/O worker #1332] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=935), Traces: [0ms] refreshing cache from master, [440ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [440ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [440ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [683ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [698ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [918ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.874 [DEBUG - New I/O worker #1332] (Deferred.java:1330) callback=retry RPC@2105020459 returned Deferred@2034735964(state=PENDING, result=null, callback=(continuation of Deferred@1011161616 after retry RPC@2105020459), errback=(continuation of Deferred@1011161616 after retry RPC@2105020459)), so the following Deferred is getting paused: Deferred@1011161616(state=PAUSED, result=Deferred@2034735964, callback=org.apache.kudu.client.AsyncKuduClient$6@3077b324 -> wakeup thread pool-3-thread-28, errback=passthrough -> wakeup thread pool-3-thread-28) 00:27:38.855 [DEBUG - New I/O worker #1344] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.875 [DEBUG - New I/O worker #1344] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=762), Traces: [0ms] refreshing cache from master, [256ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [257ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [257ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [702ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [705ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [743ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.876 [DEBUG - New I/O worker #1344] (Deferred.java:1330) callback=retry RPC@178538542 returned Deferred@245091746(state=PENDING, result=null, callback=(continuation of Deferred@1268335125 after retry RPC@178538542), errback=(continuation of Deferred@1268335125 after retry RPC@178538542)), so the following Deferred is getting paused: Deferred@1268335125(state=PAUSED, result=Deferred@245091746, callback=org.apache.kudu.client.AsyncKuduClient$6@52552882 -> wakeup thread pool-3-thread-20, errback=passthrough -> wakeup thread pool-3-thread-20) 00:27:38.850 [DEBUG - New I/O worker #1392] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.877 [DEBUG - New I/O worker #1392] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=863), Traces: [0ms] refreshing cache from master, [323ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [323ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [324ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [409ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [633ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [837ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.877 [DEBUG - New I/O worker #1392] (Deferred.java:1330) callback=retry RPC@1923702369 returned Deferred@1989592300(state=PENDING, result=null, callback=(continuation of Deferred@865358938 after retry RPC@1923702369), errback=(continuation of Deferred@865358938 after retry RPC@1923702369)), so the following Deferred is getting paused: Deferred@865358938(state=PAUSED, result=Deferred@1989592300, callback=org.apache.kudu.client.AsyncKuduClient$6@3a8f1ffb -> wakeup thread pool-3-thread-12, errback=passthrough -> wakeup thread pool-3-thread-12) 00:27:38.849 [DEBUG - New I/O worker #1365] (Negotiator.java:764) Authenticated connection [id: 0x1770a764, /127.0.0.1:53652 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.848 [DEBUG - New I/O worker #1349] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.878 [INFO - New I/O worker #1349] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:38.848 [DEBUG - New I/O worker #1375] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.882 [DEBUG - New I/O worker #1483] (Negotiator.java:764) Authenticated connection [id: 0x21d44457, /127.0.0.1:40300 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.885 [DEBUG - New I/O worker #1344] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.889 [DEBUG - New I/O worker #1332] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.890 [DEBUG - New I/O worker #1392] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.891 [DEBUG - New I/O worker #1350] (Negotiator.java:764) Authenticated connection [id: 0xe09870a4, /127.0.0.1:53664 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.895 [DEBUG - New I/O worker #1332] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.895 [DEBUG - New I/O worker #1332] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.896 [DEBUG - New I/O worker #1332] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.896 [DEBUG - New I/O worker #1332] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.896 [DEBUG - New I/O worker #1332] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.897 [DEBUG - New I/O worker #1344] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.897 [DEBUG - New I/O worker #1344] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.897 [DEBUG - New I/O worker #1524] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.898 [DEBUG - New I/O worker #1332] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.899 [DEBUG - New I/O worker #1378] (Negotiator.java:764) Authenticated connection [id: 0x605c4cd5, /127.0.0.1:53670 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.900 [DEBUG - New I/O worker #1392] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.900 [DEBUG - New I/O worker #1392] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.900 [DEBUG - New I/O worker #1392] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.901 [DEBUG - New I/O worker #1392] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.901 [DEBUG - New I/O worker #1392] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.901 [DEBUG - New I/O worker #1448] (Negotiator.java:764) Authenticated connection [id: 0x311ce61e, /127.0.0.1:53676 => /127.22.19.254:44849] using SASL/PLAIN 00:27:38.902 [DEBUG - New I/O worker #1494] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.902 [DEBUG - New I/O worker #1598] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.904 [DEBUG - New I/O worker #1392] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.905 [DEBUG - New I/O worker #1607] (Negotiator.java:764) Authenticated connection [id: 0x982ebe27, /127.0.0.1:40288 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.908 [DEBUG - New I/O worker #1500] (Negotiator.java:764) Authenticated connection [id: 0x1536d5d4, /127.0.0.1:40306 => /127.22.19.253:37879] using SASL/PLAIN 00:27:38.909 [DEBUG - New I/O worker #1346] (Negotiator.java:764) Authenticated connection [id: 0x19665ae9, /127.0.0.1:38318 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.841 [DEBUG - New I/O worker #1337] (Negotiator.java:764) Authenticated connection [id: 0x7e9ecd75, /127.0.0.1:38306 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.839 [DEBUG - New I/O worker #1340] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.912 [DEBUG - New I/O worker #1340] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.912 [DEBUG - New I/O worker #1340] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.912 [DEBUG - New I/O worker #1340] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.913 [DEBUG - New I/O worker #1340] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.913 [DEBUG - pool-3-thread-46] (Deferred.java:1330) callback=disconnect callback@2065356735 returned Deferred@245628407(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1818152876 after disconnect callback@2065356735), errback=passthrough -> (continuation of Deferred@1818152876 after disconnect callback@2065356735)), so the following Deferred is getting paused: Deferred@1818152876(state=PAUSED, result=Deferred@245628407, callback=, errback=) 00:27:38.839 [DEBUG - New I/O worker #1336] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.914 [DEBUG - New I/O worker #1336] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.914 [DEBUG - New I/O worker #1336] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.914 [DEBUG - New I/O worker #1336] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.914 [DEBUG - New I/O worker #1383] (Negotiator.java:764) Authenticated connection [id: 0x06ed276c, /127.0.0.1:38312 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.915 [DEBUG - New I/O worker #1413] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.823 [DEBUG - New I/O worker #1338] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.819 [DEBUG - New I/O worker #1374] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.915 [DEBUG - New I/O worker #1340] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.914 [DEBUG - New I/O worker #1336] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.916 [DEBUG - New I/O worker #1336] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.917 [DEBUG - New I/O worker #1346] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.917 [DEBUG - New I/O worker #1346] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=477), Traces: [0ms] refreshing cache from master, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [225ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [460ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [467ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [477ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.917 [DEBUG - New I/O worker #1364] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.914 [DEBUG - New I/O worker #1414] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.912 [DEBUG - New I/O worker #1393] (Deferred.java:1330) callback=retry RPC@1037020045 returned Deferred@972044524(state=PENDING, result=null, callback=(continuation of Deferred@2096275894 after retry RPC@1037020045), errback=(continuation of Deferred@2096275894 after retry RPC@1037020045)), so the following Deferred is getting paused: Deferred@2096275894(state=PAUSED, result=Deferred@972044524, callback=org.apache.kudu.client.AsyncKuduClient$6@d179907 -> wakeup thread pool-3-thread-26, errback=passthrough -> wakeup thread pool-3-thread-26) 00:27:38.902 [DEBUG - pool-3-thread-12] (Deferred.java:1330) callback=disconnect callback@1086040138 returned Deferred@647357684(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1534106086 after disconnect callback@1086040138), errback=passthrough -> (continuation of Deferred@1534106086 after disconnect callback@1086040138)), so the following Deferred is getting paused: Deferred@1534106086(state=PAUSED, result=Deferred@647357684, callback=, errback=) 00:27:38.897 [DEBUG - New I/O worker #1344] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.897 [DEBUG - New I/O worker #1436] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.897 [DEBUG - pool-3-thread-28] (Deferred.java:1330) callback=disconnect callback@2144211034 returned Deferred@713146136(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1673755124 after disconnect callback@2144211034), errback=passthrough -> (continuation of Deferred@1673755124 after disconnect callback@2144211034)), so the following Deferred is getting paused: Deferred@1673755124(state=PAUSED, result=Deferred@713146136, callback=, errback=) 00:27:38.891 [DEBUG - New I/O worker #1375] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.919 [DEBUG - New I/O worker #1375] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.919 [DEBUG - New I/O worker #1375] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.919 [DEBUG - New I/O worker #1375] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.919 [DEBUG - New I/O worker #1375] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.920 [DEBUG - New I/O worker #1375] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:38.920 [DEBUG - New I/O worker #1376] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.922 [DEBUG - New I/O worker #1374] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.922 [DEBUG - New I/O worker #1374] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.922 [DEBUG - New I/O worker #1374] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.922 [DEBUG - New I/O worker #1374] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.879 [DEBUG - New I/O worker #1345] (Negotiator.java:764) Authenticated connection [id: 0x8814d61d, /127.0.0.1:38324 => /127.22.19.252:32833] using SASL/PLAIN 00:27:38.922 [DEBUG - New I/O worker #1374] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.923 [DEBUG - pool-3-thread-25] (Deferred.java:1330) callback=disconnect callback@618503439 returned Deferred@159013734(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1898183976 after disconnect callback@618503439), errback=passthrough -> (continuation of Deferred@1898183976 after disconnect callback@618503439)), so the following Deferred is getting paused: Deferred@1898183976(state=PAUSED, result=Deferred@159013734, callback=, errback=) 00:27:38.923 [DEBUG - New I/O worker #1539] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.924 [DEBUG - New I/O worker #1337] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.924 [DEBUG - New I/O worker #1337] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=452), Traces: [0ms] refreshing cache from master, [185ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [185ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [186ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [346ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [418ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [452ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.925 [DEBUG - New I/O worker #1337] (Deferred.java:1330) callback=retry RPC@573620068 returned Deferred@638533626(state=PENDING, result=null, callback=(continuation of Deferred@1661842783 after retry RPC@573620068), errback=(continuation of Deferred@1661842783 after retry RPC@573620068)), so the following Deferred is getting paused: Deferred@1661842783(state=PAUSED, result=Deferred@638533626, callback=org.apache.kudu.client.AsyncKuduClient$6@7548eeb0 -> wakeup thread pool-3-thread-79, errback=passthrough -> wakeup thread pool-3-thread-79) 00:27:38.922 [DEBUG - New I/O worker #1344] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.920 [DEBUG - New I/O worker #1492] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.920 [DEBUG - pool-3-thread-42] (Deferred.java:1330) callback=disconnect callback@825875380 returned Deferred@1579900309(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1647997637 after disconnect callback@825875380), errback=passthrough -> (continuation of Deferred@1647997637 after disconnect callback@825875380)), so the following Deferred is getting paused: Deferred@1647997637(state=PAUSED, result=Deferred@1579900309, callback=, errback=) 00:27:38.925 [DEBUG - New I/O worker #1344] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.927 [DEBUG - pool-3-thread-20] (Deferred.java:1330) callback=disconnect callback@1449674422 returned Deferred@30240036(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1172513174 after disconnect callback@1449674422), errback=passthrough -> (continuation of Deferred@1172513174 after disconnect callback@1449674422)), so the following Deferred is getting paused: Deferred@1172513174(state=PAUSED, result=Deferred@30240036, callback=, errback=) 00:27:38.927 [DEBUG - New I/O worker #1398] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.928 [DEBUG - New I/O worker #1344] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.929 [DEBUG - New I/O worker #1337] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.917 [DEBUG - New I/O worker #1346] (Deferred.java:1330) callback=retry RPC@1787502789 returned Deferred@1857337676(state=PENDING, result=null, callback=(continuation of Deferred@733358846 after retry RPC@1787502789), errback=(continuation of Deferred@733358846 after retry RPC@1787502789)), so the following Deferred is getting paused: Deferred@733358846(state=PAUSED, result=Deferred@1857337676, callback=org.apache.kudu.client.AsyncKuduClient$6@2969d086 -> wakeup thread pool-3-thread-56, errback=passthrough -> wakeup thread pool-3-thread-56) 00:27:38.917 [DEBUG - New I/O worker #1363] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.939 [DEBUG - New I/O worker #1346] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.917 [DEBUG - pool-3-thread-5] (Deferred.java:1330) callback=disconnect callback@406986077 returned Deferred@223419846(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@573239795 after disconnect callback@406986077), errback=passthrough -> (continuation of Deferred@573239795 after disconnect callback@406986077)), so the following Deferred is getting paused: Deferred@573239795(state=PAUSED, result=Deferred@223419846, callback=, errback=) 00:27:38.936 [DEBUG - pool-3-thread-11] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.934 [DEBUG - New I/O worker #1337] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.940 [DEBUG - New I/O worker #1337] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.941 [DEBUG - New I/O worker #1337] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.942 [DEBUG - New I/O worker #1337] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.942 [DEBUG - New I/O worker #1337] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.943 [DEBUG - pool-3-thread-79] (Deferred.java:1330) callback=disconnect callback@1010832949 returned Deferred@745966795(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2109739981 after disconnect callback@1010832949), errback=passthrough -> (continuation of Deferred@2109739981 after disconnect callback@1010832949)), so the following Deferred is getting paused: Deferred@2109739981(state=PAUSED, result=Deferred@745966795, callback=, errback=) 00:27:38.943 [DEBUG - New I/O worker #1503] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.943 [DEBUG - New I/O worker #1365] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.945 [DEBUG - New I/O worker #1337] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.932 [DEBUG - New I/O worker #1383] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.931 [DEBUG - New I/O worker #1345] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:38.931 [DEBUG - New I/O worker #1393] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.927 [DEBUG - New I/O worker #1457] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.925 [DEBUG - pool-3-thread-17] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.924 [DEBUG - New I/O worker #1374] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.924 [DEBUG - New I/O worker #1482] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.946 [DEBUG - New I/O worker #1345] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=545), Traces: [0ms] refreshing cache from master, [391ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [391ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [391ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [501ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [507ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [530ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.950 [DEBUG - New I/O worker #1345] (Deferred.java:1330) callback=retry RPC@770343537 returned Deferred@701829908(state=PENDING, result=null, callback=(continuation of Deferred@1826064458 after retry RPC@770343537), errback=(continuation of Deferred@1826064458 after retry RPC@770343537)), so the following Deferred is getting paused: Deferred@1826064458(state=PAUSED, result=Deferred@701829908, callback=org.apache.kudu.client.AsyncKuduClient$6@247123ec -> wakeup thread pool-3-thread-40, errback=passthrough -> wakeup thread pool-3-thread-40) 00:27:38.946 [DEBUG - New I/O worker #1383] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=497), Traces: [0ms] refreshing cache from master, [212ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [213ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [213ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [459ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [463ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [484ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:38.951 [DEBUG - New I/O worker #1383] (Deferred.java:1330) callback=retry RPC@335212577 returned Deferred@398816902(state=PENDING, result=null, callback=(continuation of Deferred@1388837402 after retry RPC@335212577), errback=(continuation of Deferred@1388837402 after retry RPC@335212577)), so the following Deferred is getting paused: Deferred@1388837402(state=PAUSED, result=Deferred@398816902, callback=org.apache.kudu.client.AsyncKuduClient$6@27608f19 -> wakeup thread pool-3-thread-74, errback=passthrough -> wakeup thread pool-3-thread-74) 00:27:38.944 [DEBUG - New I/O worker #1346] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.951 [DEBUG - New I/O worker #1393] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.952 [DEBUG - New I/O worker #1393] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.952 [DEBUG - New I/O worker #1393] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.952 [DEBUG - New I/O worker #1393] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.952 [DEBUG - New I/O worker #1393] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.953 [DEBUG - New I/O worker #1345] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.953 [DEBUG - pool-3-thread-26] (Deferred.java:1330) callback=disconnect callback@499502684 returned Deferred@2005323077(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1459502893 after disconnect callback@499502684), errback=passthrough -> (continuation of Deferred@1459502893 after disconnect callback@499502684)), so the following Deferred is getting paused: Deferred@1459502893(state=PAUSED, result=Deferred@2005323077, callback=, errback=) 00:27:38.954 [DEBUG - New I/O worker #1500] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.955 [DEBUG - New I/O worker #1448] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.956 [DEBUG - New I/O worker #1393] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.951 [DEBUG - New I/O worker #1346] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.958 [DEBUG - New I/O worker #1346] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.958 [DEBUG - New I/O worker #1346] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.959 [DEBUG - New I/O worker #1346] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.959 [DEBUG - New I/O worker #1345] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.959 [DEBUG - New I/O worker #1345] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.959 [DEBUG - pool-3-thread-56] (Deferred.java:1330) callback=disconnect callback@1401932649 returned Deferred@1570722542(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1806853356 after disconnect callback@1401932649), errback=passthrough -> (continuation of Deferred@1806853356 after disconnect callback@1401932649)), so the following Deferred is getting paused: Deferred@1806853356(state=PAUSED, result=Deferred@1570722542, callback=, errback=) 00:27:38.959 [DEBUG - New I/O worker #1350] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.960 [DEBUG - New I/O worker #1502] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.960 [DEBUG - New I/O worker #1345] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.960 [DEBUG - New I/O worker #1345] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.960 [DEBUG - New I/O worker #1345] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.961 [DEBUG - New I/O worker #1346] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.961 [DEBUG - New I/O worker #1383] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:38.962 [DEBUG - pool-3-thread-40] (Deferred.java:1330) callback=disconnect callback@1652967501 returned Deferred@1190294416(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2136037386 after disconnect callback@1652967501), errback=passthrough -> (continuation of Deferred@2136037386 after disconnect callback@1652967501)), so the following Deferred is getting paused: Deferred@2136037386(state=PAUSED, result=Deferred@1190294416, callback=, errback=) 00:27:38.963 [DEBUG - New I/O worker #1378] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.963 [DEBUG - New I/O worker #1483] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.965 [DEBUG - New I/O worker #1345] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.966 [DEBUG - New I/O worker #1383] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:38.966 [DEBUG - New I/O worker #1383] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:38.966 [DEBUG - New I/O worker #1383] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:38.966 [DEBUG - New I/O worker #1383] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:38.967 [DEBUG - New I/O worker #1383] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:38.967 [DEBUG - New I/O worker #1607] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:38.968 [DEBUG - New I/O worker #1497] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:38.968 [DEBUG - pool-3-thread-74] (Deferred.java:1330) callback=disconnect callback@1532624184 returned Deferred@454598417(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@650629672 after disconnect callback@1532624184), errback=passthrough -> (continuation of Deferred@650629672 after disconnect callback@1532624184)), so the following Deferred is getting paused: Deferred@650629672(state=PAUSED, result=Deferred@454598417, callback=, errback=) 00:27:38.968 [DEBUG - New I/O worker #1383] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:38.984 [DEBUG - pool-3-thread-44] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.986 [DEBUG - pool-3-thread-48] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:38.992 [DEBUG - New I/O worker #1798] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.993 [DEBUG - New I/O worker #1811] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:38.997 [DEBUG - New I/O worker #1808] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.001 [DEBUG - New I/O worker #1796] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.008 [DEBUG - New I/O worker #1810] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.011 [DEBUG - New I/O worker #1798] (Negotiator.java:764) Authenticated connection [id: 0x80ccba08, /127.0.0.1:40312 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.014 [DEBUG - New I/O worker #1796] (Negotiator.java:764) Authenticated connection [id: 0x7b8603da, /127.0.0.1:53682 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.020 [DEBUG - New I/O worker #1795] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.037 [DEBUG - New I/O worker #1811] (Negotiator.java:764) Authenticated connection [id: 0x97c8698c, /127.0.0.1:40318 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.044 [DEBUG - New I/O worker #1810] (Negotiator.java:764) Authenticated connection [id: 0x54402cf3, /127.0.0.1:53688 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.047 [DEBUG - New I/O worker #1795] (Negotiator.java:764) Authenticated connection [id: 0x3143f272, /127.0.0.1:38336 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.050 [DEBUG - New I/O worker #1808] (Negotiator.java:764) Authenticated connection [id: 0x77530b0d, /127.0.0.1:38342 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.052 [DEBUG - New I/O worker #1795] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.053 [DEBUG - New I/O worker #1795] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=67), Traces: [0ms] refreshing cache from master, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [29ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [33ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [65ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.053 [DEBUG - New I/O worker #1808] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.053 [DEBUG - New I/O worker #1808] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=66), Traces: [0ms] refreshing cache from master, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [54ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [61ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [66ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.053 [DEBUG - New I/O worker #1795] (Deferred.java:1330) callback=retry RPC@1571759047 returned Deferred@1502657706(state=PENDING, result=null, callback=(continuation of Deferred@479341052 after retry RPC@1571759047), errback=(continuation of Deferred@479341052 after retry RPC@1571759047)), so the following Deferred is getting paused: Deferred@479341052(state=PAUSED, result=Deferred@1502657706, callback=org.apache.kudu.client.AsyncKuduClient$6@951ebb3 -> wakeup thread pool-3-thread-44, errback=passthrough -> wakeup thread pool-3-thread-44) 00:27:39.056 [DEBUG - New I/O worker #1795] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.058 [DEBUG - New I/O worker #1808] (Deferred.java:1330) callback=retry RPC@589498485 returned Deferred@656185545(state=PENDING, result=null, callback=(continuation of Deferred@1646133838 after retry RPC@589498485), errback=(continuation of Deferred@1646133838 after retry RPC@589498485)), so the following Deferred is getting paused: Deferred@1646133838(state=PAUSED, result=Deferred@656185545, callback=org.apache.kudu.client.AsyncKuduClient$6@686d5c8f -> wakeup thread pool-3-thread-48, errback=passthrough -> wakeup thread pool-3-thread-48) 00:27:39.059 [DEBUG - New I/O worker #1808] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.062 [DEBUG - New I/O worker #1795] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.062 [DEBUG - New I/O worker #1808] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.063 [DEBUG - New I/O worker #1795] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.063 [DEBUG - New I/O worker #1795] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.063 [DEBUG - New I/O worker #1795] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.064 [DEBUG - New I/O worker #1795] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.063 [DEBUG - New I/O worker #1808] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.064 [DEBUG - New I/O worker #1808] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.064 [DEBUG - New I/O worker #1808] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.065 [DEBUG - New I/O worker #1796] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:39.065 [DEBUG - New I/O worker #1808] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.065 [DEBUG - pool-3-thread-48] (Deferred.java:1330) callback=disconnect callback@1526399265 returned Deferred@1390219889(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@281182961 after disconnect callback@1526399265), errback=passthrough -> (continuation of Deferred@281182961 after disconnect callback@1526399265)), so the following Deferred is getting paused: Deferred@281182961(state=PAUSED, result=Deferred@1390219889, callback=, errback=) 00:27:39.064 [DEBUG - pool-3-thread-44] (Deferred.java:1330) callback=disconnect callback@630208617 returned Deferred@1209040305(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1740490104 after disconnect callback@630208617), errback=passthrough -> (continuation of Deferred@1740490104 after disconnect callback@630208617)), so the following Deferred is getting paused: Deferred@1740490104(state=PAUSED, result=Deferred@1209040305, callback=, errback=) 00:27:39.065 [DEBUG - New I/O worker #1798] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:39.067 [DEBUG - New I/O worker #1811] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:39.067 [DEBUG - New I/O worker #1795] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:39.068 [DEBUG - New I/O worker #1810] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:39.068 [DEBUG - New I/O worker #1808] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:39.077 [DEBUG - pool-3-thread-3] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.092 [DEBUG - pool-3-thread-43] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.099 [DEBUG - pool-3-thread-30] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.100 [DEBUG - pool-3-thread-21] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.108 [DEBUG - New I/O worker #1919] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.109 [DEBUG - New I/O worker #1915] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.112 [DEBUG - New I/O worker #1918] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.125 [DEBUG - New I/O worker #1889] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.132 [DEBUG - New I/O worker #1915] (Negotiator.java:764) Authenticated connection [id: 0x3692270a, /127.0.0.1:38348 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.137 [DEBUG - New I/O worker #1915] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.137 [DEBUG - New I/O worker #1915] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=38), Traces: [0ms] refreshing cache from master, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [38ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.138 [DEBUG - New I/O worker #1915] (Deferred.java:1330) callback=retry RPC@186464129 returned Deferred@253932090(state=PENDING, result=null, callback=(continuation of Deferred@1243626938 after retry RPC@186464129), errback=(continuation of Deferred@1243626938 after retry RPC@186464129)), so the following Deferred is getting paused: Deferred@1243626938(state=PAUSED, result=Deferred@253932090, callback=org.apache.kudu.client.AsyncKuduClient$6@1963ad3d -> wakeup thread pool-3-thread-30, errback=passthrough -> wakeup thread pool-3-thread-30) 00:27:39.139 [DEBUG - New I/O worker #1918] (Negotiator.java:764) Authenticated connection [id: 0x8d90a867, /127.0.0.1:53694 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.140 [DEBUG - New I/O worker #1915] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.144 [DEBUG - New I/O worker #1915] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.144 [DEBUG - New I/O worker #1915] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.144 [DEBUG - New I/O worker #1915] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.144 [DEBUG - New I/O worker #1915] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.145 [DEBUG - New I/O worker #1915] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:39.147456 24969 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:39.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.104319 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40324 00:27:39.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.104759 (+ 440us) server_negotiation.cc:184] Beginning negotiation 00:27:39.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.104783 (+ 24us) server_negotiation.cc:373] Waiting for connection header 00:27:39.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.105643 (+ 860us) server_negotiation.cc:381] Connection header received 00:27:39.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.105899 (+ 256us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:39.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.105923 (+ 24us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:39.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.106050 (+ 127us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:39.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.106184 (+ 134us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:39.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.109759 (+ 3575us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:39.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.110899 (+ 1140us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:39.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.146462 (+ 35563us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40324: BlockingRecv error: recv got EOF from 127.0.0.1:40324 (error 108) 00:27:39.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":152} 00:27:39.149 [DEBUG - New I/O worker #1915] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.149 [DEBUG - New I/O worker #1918] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:39.157 [DEBUG - New I/O worker #1888] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.157 [DEBUG - New I/O worker #1889] (Negotiator.java:764) Authenticated connection [id: 0xe9e4c5ec, /127.0.0.1:40330 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.165 [DEBUG - New I/O worker #1886] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.178 [DEBUG - New I/O worker #1886] (Negotiator.java:764) Authenticated connection [id: 0x894d5057, /127.0.0.1:38354 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.181 [DEBUG - New I/O worker #1888] (Negotiator.java:764) Authenticated connection [id: 0x85b54541, /127.0.0.1:53700 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.181 [INFO - New I/O worker #1919] (Connection.java:435) [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:39.183 [DEBUG - New I/O worker #1919] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:39.184 [INFO - New I/O worker #1919] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:39.185 [DEBUG - New I/O worker #1886] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.185 [DEBUG - New I/O worker #1886] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=93), Traces: [0ms] refreshing cache from master, [27ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [28ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [28ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [68ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [91ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [92ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.188 [DEBUG - New I/O worker #1886] (Deferred.java:1330) callback=retry RPC@1106349856 returned Deferred@1171133382(state=PENDING, result=null, callback=(continuation of Deferred@13403419 after retry RPC@1106349856), errback=(continuation of Deferred@13403419 after retry RPC@1106349856)), so the following Deferred is getting paused: Deferred@13403419(state=PAUSED, result=Deferred@1171133382, callback=org.apache.kudu.client.AsyncKuduClient$6@7cc3ae7c -> wakeup thread pool-3-thread-43, errback=passthrough -> wakeup thread pool-3-thread-43) 00:27:39.189 [DEBUG - New I/O worker #1886] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.192 [DEBUG - New I/O worker #1886] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.193 [DEBUG - New I/O worker #1886] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.193 [DEBUG - New I/O worker #1886] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.193 [DEBUG - New I/O worker #1886] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.193 [DEBUG - New I/O worker #1886] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.194 [DEBUG - New I/O worker #1888] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:39.195 [DEBUG - pool-3-thread-43] (Deferred.java:1330) callback=disconnect callback@792191656 returned Deferred@28593609(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2093187112 after disconnect callback@792191656), errback=passthrough -> (continuation of Deferred@2093187112 after disconnect callback@792191656)), so the following Deferred is getting paused: Deferred@2093187112(state=PAUSED, result=Deferred@28593609, callback=, errback=) 00:27:39.195 [DEBUG - New I/O worker #1889] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:39.197 [DEBUG - New I/O worker #1886] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:39.218 [DEBUG - New I/O worker #1869] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.219 [DEBUG - New I/O worker #1872] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.221 [DEBUG - New I/O worker #1871] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.234 [DEBUG - New I/O worker #1872] (Negotiator.java:764) Authenticated connection [id: 0x9f56653e, /127.0.0.1:40336 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.239 [DEBUG - New I/O worker #1871] (Negotiator.java:764) Authenticated connection [id: 0x4bf01ebb, /127.0.0.1:53706 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.241 [DEBUG - New I/O worker #1869] (Negotiator.java:764) Authenticated connection [id: 0x05765508, /127.0.0.1:38360 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.244 [DEBUG - New I/O worker #1869] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.244 [DEBUG - New I/O worker #1869] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=166), Traces: [0ms] refreshing cache from master, [138ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [138ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [139ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [160ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [165ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [166ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.246 [DEBUG - New I/O worker #1869] (Deferred.java:1330) callback=retry RPC@1581345941 returned Deferred@1518265221(state=PENDING, result=null, callback=(continuation of Deferred@528246446 after retry RPC@1581345941), errback=(continuation of Deferred@528246446 after retry RPC@1581345941)), so the following Deferred is getting paused: Deferred@528246446(state=PAUSED, result=Deferred@1518265221, callback=org.apache.kudu.client.AsyncKuduClient$6@7b598494 -> wakeup thread pool-3-thread-3, errback=passthrough -> wakeup thread pool-3-thread-3) 00:27:39.247 [DEBUG - New I/O worker #1869] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.251 [DEBUG - New I/O worker #1869] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.251 [DEBUG - New I/O worker #1869] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.251 [DEBUG - New I/O worker #1869] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.251 [DEBUG - New I/O worker #1869] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.252 [DEBUG - New I/O worker #1869] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.252 [DEBUG - pool-3-thread-3] (Deferred.java:1330) callback=disconnect callback@360944027 returned Deferred@821533069(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@796593436 after disconnect callback@360944027), errback=passthrough -> (continuation of Deferred@796593436 after disconnect callback@360944027)), so the following Deferred is getting paused: Deferred@796593436(state=PAUSED, result=Deferred@821533069, callback=, errback=) 00:27:39.253 [DEBUG - New I/O worker #1869] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.253 [DEBUG - New I/O worker #1872] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:39.254 [DEBUG - New I/O worker #1871] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:39.254 [DEBUG - New I/O worker #1795] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.255 [DEBUG - pool-3-thread-62] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.256 [DEBUG - New I/O worker #1808] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.258 [DEBUG - pool-3-thread-66] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.260 [DEBUG - pool-3-thread-1] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.262 [DEBUG - pool-3-thread-38] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.283 [DEBUG - pool-3-thread-31] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.299 [DEBUG - pool-3-thread-9] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.325 [DEBUG - pool-3-thread-22] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.327 [DEBUG - New I/O worker #1383] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.331 [DEBUG - New I/O worker #1345] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.332 [DEBUG - New I/O worker #1346] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.335 [DEBUG - New I/O worker #1393] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.337 [DEBUG - pool-3-thread-8] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.339 [DEBUG - New I/O worker #1374] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.340 [DEBUG - pool-3-thread-49] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.343 [DEBUG - New I/O worker #1337] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.345 [DEBUG - pool-3-thread-33] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.364 [DEBUG - New I/O worker #1344] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.367 [DEBUG - New I/O worker #1376] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:39.368 [DEBUG - New I/O worker #1340] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.379 [DEBUG - New I/O worker #1392] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.391 [DEBUG - New I/O worker #1332] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.392 [DEBUG - New I/O worker #1570] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:39.395 [DEBUG - New I/O worker #1370] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:39.423 [DEBUG - New I/O worker #1871] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:39.422 [DEBUG - New I/O worker #1886] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.422 [DEBUG - pool-3-thread-30] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:39.411 [DEBUG - New I/O worker #1364] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:39.411 [DEBUG - New I/O worker #1338] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.407 [DEBUG - New I/O worker #1385] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:39.439 [INFO - New I/O worker #1385] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:39.403 [DEBUG - New I/O worker #1387] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:39.399 [DEBUG - New I/O worker #1445] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:39.439 [INFO - New I/O worker #1445] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:39.440 [DEBUG - New I/O worker #1844] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.441 [DEBUG - New I/O worker #2014] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.442 [DEBUG - New I/O worker #1847] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.439 [DEBUG - New I/O worker #1877] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.451 [DEBUG - New I/O worker #1846] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.452 [DEBUG - pool-3-thread-7] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.453 [DEBUG - New I/O worker #1838] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.453 [DEBUG - New I/O worker #1786] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.457 [DEBUG - New I/O worker #1788] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.460 [DEBUG - New I/O worker #2050] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.463 [DEBUG - New I/O worker #1777] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.464 [DEBUG - New I/O worker #1835] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.464 [DEBUG - New I/O worker #1809] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.466 [DEBUG - New I/O worker #2012] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.467 [DEBUG - New I/O worker #1837] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.468 [DEBUG - New I/O worker #1769] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.470 [DEBUG - New I/O worker #1778] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.472 [DEBUG - New I/O worker #1770] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.485 [DEBUG - New I/O worker #1699] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.486 [DEBUG - New I/O worker #1861] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.495 [DEBUG - pool-3-thread-61] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.502 [DEBUG - pool-3-thread-47] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.503 [DEBUG - pool-3-thread-16] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.505 [DEBUG - New I/O worker #2014] (Negotiator.java:764) Authenticated connection [id: 0xa9e8eca0, /127.0.0.1:40358 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.510 [DEBUG - pool-3-thread-50] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.511 [DEBUG - pool-3-thread-41] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.513 [DEBUG - New I/O worker #1730] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.518 [DEBUG - New I/O worker #1754] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.519 [DEBUG - New I/O worker #1727] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.519 [DEBUG - New I/O worker #1697] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.520 [DEBUG - New I/O worker #1752] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.521 [DEBUG - New I/O worker #1676] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.528 [DEBUG - New I/O worker #1677] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.532 [DEBUG - New I/O worker #2050] (Negotiator.java:764) Authenticated connection [id: 0xcc2b952f, /127.0.0.1:53748 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.533 [DEBUG - New I/O worker #1678] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.534 [DEBUG - New I/O worker #1862] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.535 [DEBUG - New I/O worker #1827] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.536 [DEBUG - New I/O worker #2051] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.537 [DEBUG - New I/O worker #1860] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.548 [DEBUG - New I/O worker #1828] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.550 [DEBUG - New I/O worker #1771] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.551 [DEBUG - New I/O worker #1789] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.552 [DEBUG - New I/O worker #1700] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.552 [DEBUG - New I/O worker #1753] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.553 [DEBUG - New I/O worker #1729] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.553 [DEBUG - New I/O worker #1829] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.554 [DEBUG - New I/O worker #1779] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.555 [DEBUG - New I/O worker #1699] (Negotiator.java:764) Authenticated connection [id: 0x9d9bd7b9, /127.0.0.1:53712 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.562 [DEBUG - New I/O worker #2012] (Negotiator.java:764) Authenticated connection [id: 0x3cf9139a, /127.0.0.1:53726 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.563 [DEBUG - New I/O worker #1809] (Negotiator.java:764) Authenticated connection [id: 0x00b048b7, /127.0.0.1:38402 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.568 [DEBUG - New I/O worker #1809] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.569 [DEBUG - New I/O worker #1809] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=269), Traces: [0ms] refreshing cache from master, [128ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [128ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [128ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [238ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [269ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.570 [DEBUG - New I/O worker #1754] (Negotiator.java:764) Authenticated connection [id: 0x4842c967, /127.0.0.1:40400 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.571 [DEBUG - New I/O worker #1809] (Deferred.java:1330) callback=retry RPC@699851882 returned Deferred@763983236(state=PENDING, result=null, callback=(continuation of Deferred@1754000977 after retry RPC@699851882), errback=(continuation of Deferred@1754000977 after retry RPC@699851882)), so the following Deferred is getting paused: Deferred@1754000977(state=PAUSED, result=Deferred@763983236, callback=org.apache.kudu.client.AsyncKuduClient$6@47a2cb0a -> wakeup thread pool-3-thread-9, errback=passthrough -> wakeup thread pool-3-thread-9) 00:27:39.574 [DEBUG - New I/O worker #1809] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.578 [DEBUG - New I/O worker #1809] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.578 [DEBUG - New I/O worker #1809] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.578 [DEBUG - New I/O worker #1809] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.578 [DEBUG - New I/O worker #1809] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.579 [DEBUG - New I/O worker #1809] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.579 [DEBUG - New I/O worker #1730] (Negotiator.java:764) Authenticated connection [id: 0x27d7a7c3, /127.0.0.1:40348 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.579 [DEBUG - New I/O worker #2050] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:39.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:39.581913 26092 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:39.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.447104 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40378 00:27:39.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.489156 (+ 42052us) server_negotiation.cc:184] Beginning negotiation 00:27:39.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.489196 (+ 40us) server_negotiation.cc:373] Waiting for connection header 00:27:39.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.489240 (+ 44us) server_negotiation.cc:381] Connection header received 00:27:39.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.489336 (+ 96us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:39.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.489360 (+ 24us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:39.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.489485 (+ 125us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:39.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.491154 (+ 1669us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:39.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.562383 (+ 71229us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:39.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.564929 (+ 2546us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:39.582 [DEBUG - New I/O worker #1809] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.580693 (+ 15764us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40378: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) 00:27:39.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":41738,"thread_start_us":36150,"threads_started":1} 00:27:39.586 [DEBUG - New I/O worker #1877] (Negotiator.java:764) Authenticated connection [id: 0xacbf1c3d, /127.0.0.1:38378 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.588 [DEBUG - New I/O worker #1877] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.589 [DEBUG - New I/O worker #1877] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=332), Traces: [0ms] refreshing cache from master, [167ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [167ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [167ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [252ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [310ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [332ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.589 [DEBUG - New I/O worker #1877] (Deferred.java:1330) callback=retry RPC@617921231 returned Deferred@552282765(state=PENDING, result=null, callback=(continuation of Deferred@1709814004 after retry RPC@617921231), errback=(continuation of Deferred@1709814004 after retry RPC@617921231)), so the following Deferred is getting paused: Deferred@1709814004(state=PAUSED, result=Deferred@552282765, callback=org.apache.kudu.client.AsyncKuduClient$6@5545191d -> wakeup thread pool-3-thread-62, errback=passthrough -> wakeup thread pool-3-thread-62) 00:27:39.590 [DEBUG - New I/O worker #1877] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.593 [DEBUG - New I/O worker #1877] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.593 [DEBUG - New I/O worker #1877] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.593 [DEBUG - New I/O worker #1877] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.593 [DEBUG - New I/O worker #1877] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.594 [DEBUG - New I/O worker #1877] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.595 [DEBUG - New I/O worker #1677] (Negotiator.java:764) Authenticated connection [id: 0x781a09a5, /127.0.0.1:53776 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.598 [DEBUG - New I/O worker #1676] (Negotiator.java:764) Authenticated connection [id: 0x33972373, /127.0.0.1:38430 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.599 [DEBUG - New I/O worker #1678] (Negotiator.java:764) Authenticated connection [id: 0xa3cd22d5, /127.0.0.1:40406 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.601 [DEBUG - New I/O worker #1676] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.601 [DEBUG - New I/O worker #1676] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=734), Traces: [0ms] refreshing cache from master, [567ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [567ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [568ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [731ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [734ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.602 [DEBUG - New I/O worker #1676] (Deferred.java:1330) callback=retry RPC@303764225 returned Deferred@371492250(state=PENDING, result=null, callback=(continuation of Deferred@1395005754 after retry RPC@303764225), errback=(continuation of Deferred@1395005754 after retry RPC@303764225)), so the following Deferred is getting paused: Deferred@1395005754(state=PAUSED, result=Deferred@371492250, callback=org.apache.kudu.client.AsyncKuduClient$6@2876160a -> wakeup thread pool-3-thread-37, errback=passthrough -> wakeup thread pool-3-thread-37) 00:27:39.604 [DEBUG - New I/O worker #1676] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.613 [DEBUG - New I/O worker #1676] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.614 [DEBUG - New I/O worker #1676] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.614 [DEBUG - New I/O worker #1676] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.615 [DEBUG - New I/O worker #1676] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.616 [DEBUG - New I/O worker #1676] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299998, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.617 [DEBUG - pool-3-thread-37] (Deferred.java:1330) callback=disconnect callback@1986058889 returned Deferred@85033665(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@551387482 after disconnect callback@1986058889), errback=passthrough -> (continuation of Deferred@551387482 after disconnect callback@1986058889)), so the following Deferred is getting paused: Deferred@551387482(state=PAUSED, result=Deferred@85033665, callback=, errback=) 00:27:39.617 [DEBUG - New I/O worker #1678] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:39.618 [DEBUG - New I/O worker #1676] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.621 [DEBUG - New I/O worker #1727] (Negotiator.java:764) Authenticated connection [id: 0x2adcf14a, /127.0.0.1:38372 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.623 [DEBUG - New I/O worker #1752] (Negotiator.java:764) Authenticated connection [id: 0xd0fd7cc7, /127.0.0.1:38424 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.623 [DEBUG - New I/O worker #1727] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.624 [DEBUG - New I/O worker #1727] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=683), Traces: [0ms] refreshing cache from master, [407ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [407ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [408ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [643ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [683ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.624 [DEBUG - New I/O worker #1727] (Deferred.java:1330) callback=retry RPC@595758982 returned Deferred@666701614(state=PENDING, result=null, callback=(continuation of Deferred@1656718781 after retry RPC@595758982), errback=(continuation of Deferred@1656718781 after retry RPC@595758982)), so the following Deferred is getting paused: Deferred@1656718781(state=PAUSED, result=Deferred@666701614, callback=org.apache.kudu.client.AsyncKuduClient$6@645e7a20 -> wakeup thread pool-3-thread-11, errback=passthrough -> wakeup thread pool-3-thread-11) 00:27:39.626 [DEBUG - New I/O worker #1727] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.628 [DEBUG - New I/O worker #1677] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:39.630 [DEBUG - New I/O worker #1727] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.630 [DEBUG - New I/O worker #1770] (Negotiator.java:764) Authenticated connection [id: 0x78ac6833, /127.0.0.1:53766 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.630 [DEBUG - New I/O worker #1727] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.631 [DEBUG - New I/O worker #1727] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.631 [DEBUG - New I/O worker #1727] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.631 [DEBUG - pool-3-thread-65] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.631 [DEBUG - New I/O worker #1727] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.632 [DEBUG - pool-3-thread-62] (Deferred.java:1330) callback=disconnect callback@330300596 returned Deferred@1141977900(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1351410029 after disconnect callback@330300596), errback=passthrough -> (continuation of Deferred@1351410029 after disconnect callback@330300596)), so the following Deferred is getting paused: Deferred@1351410029(state=PAUSED, result=Deferred@1141977900, callback=, errback=) 00:27:39.633 [DEBUG - New I/O worker #2014] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:39.633900 25051 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.348719 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53718 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.349149 (+ 430us) server_negotiation.cc:184] Beginning negotiation 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.349184 (+ 35us) server_negotiation.cc:373] Waiting for connection header 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.514605 (+165421us) server_negotiation.cc:381] Connection header received 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.514731 (+ 126us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.514755 (+ 24us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.514875 (+ 120us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.515053 (+ 178us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.598363 (+ 83310us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.602991 (+ 4628us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.632811 (+ 29820us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53718: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) 00:27:39.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":144} 00:27:39.634 [DEBUG - New I/O worker #2012] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:39.635 [DEBUG - New I/O worker #1877] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:39.635 [DEBUG - New I/O worker #1727] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.636 [DEBUG - New I/O worker #1697] (Negotiator.java:764) Authenticated connection [id: 0xd0720ed9, /127.0.0.1:38366 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.639 [DEBUG - New I/O worker #1752] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.639 [DEBUG - New I/O worker #1752] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=293), Traces: [0ms] refreshing cache from master, [88ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [88ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [88ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [233ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [294ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.642 [DEBUG - New I/O worker #1752] (Deferred.java:1330) callback=retry RPC@1616880981 returned Deferred@1684028806(state=PENDING, result=null, callback=(continuation of Deferred@560115566 after retry RPC@1616880981), errback=(continuation of Deferred@560115566 after retry RPC@1616880981)), so the following Deferred is getting paused: Deferred@560115566(state=PAUSED, result=Deferred@1684028806, callback=org.apache.kudu.client.AsyncKuduClient$6@de17252 -> wakeup thread pool-3-thread-33, errback=passthrough -> wakeup thread pool-3-thread-33) 00:27:39.642 [DEBUG - New I/O worker #1752] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.646 [DEBUG - New I/O worker #1752] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.646 [DEBUG - New I/O worker #1752] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.646 [DEBUG - New I/O worker #1752] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.647 [DEBUG - New I/O worker #1752] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.647 [DEBUG - New I/O worker #1752] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.648 [DEBUG - New I/O worker #1838] (Negotiator.java:764) Authenticated connection [id: 0x5a854eb4, /127.0.0.1:40366 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:39.650127 26106 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.473103 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53770 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.474260 (+ 1157us) server_negotiation.cc:184] Beginning negotiation 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.474298 (+ 38us) server_negotiation.cc:373] Waiting for connection header 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.526990 (+ 52692us) server_negotiation.cc:381] Connection header received 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.527101 (+ 111us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.527125 (+ 24us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.527258 (+ 133us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.527422 (+ 164us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.603538 (+ 76116us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.604651 (+ 1113us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.649104 (+ 44453us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53770: BlockingRecv error: recv got EOF from 127.0.0.1:53770 (error 108) 00:27:39.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":593,"threads_started":1} 00:27:39.652 [DEBUG - New I/O worker #1752] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.652 [DEBUG - New I/O worker #1697] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.652 [DEBUG - New I/O worker #1697] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=705), Traces: [0ms] refreshing cache from master, [394ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [394ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [394ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [612ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [691ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.653 [DEBUG - New I/O worker #1697] (Deferred.java:1330) callback=retry RPC@1289335603 returned Deferred@1223044720(state=PENDING, result=null, callback=(continuation of Deferred@233093384 after retry RPC@1289335603), errback=(continuation of Deferred@233093384 after retry RPC@1289335603)), so the following Deferred is getting paused: Deferred@233093384(state=PAUSED, result=Deferred@1223044720, callback=org.apache.kudu.client.AsyncKuduClient$6@3d62dbf1 -> wakeup thread pool-3-thread-17, errback=passthrough -> wakeup thread pool-3-thread-17) 00:27:39.653 [DEBUG - New I/O worker #1904] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.655 [DEBUG - New I/O worker #1697] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.657 [DEBUG - New I/O worker #1907] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.659 [DEBUG - New I/O worker #1906] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.661 [DEBUG - New I/O worker #1846] (Negotiator.java:764) Authenticated connection [id: 0x3276e9e9, /127.0.0.1:53728 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.664 [DEBUG - New I/O worker #1697] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.664 [DEBUG - New I/O worker #1697] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.665 [DEBUG - New I/O worker #1697] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.665 [DEBUG - New I/O worker #1697] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.665 [DEBUG - New I/O worker #1697] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.666 [DEBUG - New I/O worker #1699] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:39.670 [DEBUG - New I/O worker #1788] (Negotiator.java:764) Authenticated connection [id: 0x0d5c4b13, /127.0.0.1:53754 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:39.668908 25010 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.343050 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40342 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.343540 (+ 490us) server_negotiation.cc:184] Beginning negotiation 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.343564 (+ 24us) server_negotiation.cc:373] Waiting for connection header 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.513641 (+170077us) server_negotiation.cc:381] Connection header received 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.513757 (+ 116us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.513788 (+ 31us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.513913 (+ 125us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.514064 (+ 151us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.601817 (+ 87753us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.602618 (+ 801us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.667248 (+ 64630us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40342: BlockingRecv error: recv got EOF from 127.0.0.1:40342 (error 108) 00:27:39.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":183} 00:27:39.674 [DEBUG - New I/O worker #1697] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.682 [DEBUG - New I/O worker #1844] (Negotiator.java:764) Authenticated connection [id: 0x2a66aedc, /127.0.0.1:38380 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.696 [DEBUG - New I/O worker #1844] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.697 [DEBUG - New I/O worker #1844] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=436), Traces: [0ms] refreshing cache from master, [163ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [163ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [163ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [429ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [436ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.697 [DEBUG - New I/O worker #1844] (Deferred.java:1330) callback=retry RPC@155840337 returned Deferred@225864352(state=PENDING, result=null, callback=(continuation of Deferred@1215620458 after retry RPC@155840337), errback=(continuation of Deferred@1215620458 after retry RPC@155840337)), so the following Deferred is getting paused: Deferred@1215620458(state=PAUSED, result=Deferred@225864352, callback=org.apache.kudu.client.AsyncKuduClient$6@3f03ed35 -> wakeup thread pool-3-thread-1, errback=passthrough -> wakeup thread pool-3-thread-1) 00:27:39.700 [DEBUG - New I/O worker #1844] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.708 [DEBUG - New I/O worker #1844] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.708 [DEBUG - New I/O worker #1844] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.708 [DEBUG - New I/O worker #1844] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.709 [DEBUG - New I/O worker #1844] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.709 [DEBUG - New I/O worker #1844] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.710 [DEBUG - New I/O worker #1846] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:39.710 [DEBUG - New I/O worker #1844] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:39.714272 26084 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:39.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.427532 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40360 00:27:39.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.436332 (+ 8800us) server_negotiation.cc:184] Beginning negotiation 00:27:39.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.436373 (+ 41us) server_negotiation.cc:373] Waiting for connection header 00:27:39.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.436444 (+ 71us) server_negotiation.cc:381] Connection header received 00:27:39.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.436574 (+ 130us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:39.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.436600 (+ 26us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:39.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.436722 (+ 122us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:39.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.444550 (+ 7828us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:39.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.483262 (+ 38712us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:39.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.493607 (+ 10345us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:39.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.670836 (+177229us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:39.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.675794 (+ 4958us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:39.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.676026 (+ 232us) server_negotiation.cc:589] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 00:27:39.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:39.712401 (+ 36375us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40360: BlockingRecv error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) 00:27:39.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":3248,"server-negotiator.queue_time_us":7771,"thread_start_us":5296,"threads_started":1} 00:27:39.727 [DEBUG - New I/O worker #1837] (Negotiator.java:764) Authenticated connection [id: 0xb83d1e60, /127.0.0.1:53736 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.731 [INFO - New I/O worker #1700] (Connection.java:435) [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:39.731 [DEBUG - New I/O worker #1700] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:39.899 [INFO - New I/O worker #1700] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:39.898 [DEBUG - New I/O worker #1906] (Negotiator.java:764) Authenticated connection [id: 0x9da6c722, /127.0.0.1:53790 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.902 [DEBUG - New I/O worker #1904] (Negotiator.java:764) Authenticated connection [id: 0x59c87db7, /127.0.0.1:38444 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.898 [DEBUG - New I/O worker #1829] (Negotiator.java:764) Authenticated connection [id: 0xfaf3c11b, /127.0.0.1:40372 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.905 [DEBUG - New I/O worker #1904] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.895 [DEBUG - New I/O worker #1934] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.894 [DEBUG - New I/O worker #2] (Connection.java:429) [peer master-127.22.19.254:44849(127.22.19.254:44849)] encountered a read timeout; closing the channel 00:27:39.906 [DEBUG - New I/O worker #2] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: [peer master-127.22.19.254:44849(127.22.19.254:44849)] encountered a read timeout; closing the channel 00:27:39.894 [DEBUG - New I/O worker #3] (Connection.java:429) [peer master-127.22.19.253:37879(127.22.19.253:37879)] encountered a read timeout; closing the channel 00:27:39.906 [DEBUG - New I/O worker #3] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: [peer master-127.22.19.253:37879(127.22.19.253:37879)] encountered a read timeout; closing the channel 00:27:39.893 [DEBUG - New I/O worker #1935] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.893 [DEBUG - New I/O worker #1914] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.884 [DEBUG - pool-3-thread-27] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.869 [DEBUG - pool-3-thread-59] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.867 [DEBUG - pool-3-thread-58] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.867 [DEBUG - pool-3-thread-72] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.852 [DEBUG - pool-3-thread-57] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.909 [DEBUG - New I/O worker #1904] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=808), Traces: [0ms] refreshing cache from master, [514ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [514ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [514ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [803ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [804ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.844 [DEBUG - pool-3-thread-39] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.840 [DEBUG - pool-3-thread-77] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.820 [DEBUG - Time-limited test] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.819 [DEBUG - pool-3-thread-68] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.818 [DEBUG - pool-3-thread-54] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.817 [DEBUG - pool-3-thread-23] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.817 [DEBUG - pool-3-thread-75] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.816 [DEBUG - pool-3-thread-51] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.815 [DEBUG - pool-3-thread-52] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.813 [DEBUG - pool-3-thread-71] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.799 [DEBUG - New I/O worker #1912] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.797 [DEBUG - New I/O worker #1943] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.911 [DEBUG - New I/O worker #1904] (Deferred.java:1330) callback=retry RPC@701989599 returned Deferred@770179901(state=PENDING, result=null, callback=(continuation of Deferred@1760200932 after retry RPC@701989599), errback=(continuation of Deferred@1760200932 after retry RPC@701989599)), so the following Deferred is getting paused: Deferred@1760200932(state=PAUSED, result=Deferred@770179901, callback=org.apache.kudu.client.AsyncKuduClient$6@4c9cd7b4 -> wakeup thread pool-3-thread-21, errback=passthrough -> wakeup thread pool-3-thread-21) 00:27:39.797 [DEBUG - New I/O worker #1720] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.795 [DEBUG - New I/O worker #1719] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.795 [DEBUG - New I/O worker #1718] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.793 [DEBUG - New I/O worker #1942] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.788 [DEBUG - New I/O worker #1712] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.787 [DEBUG - New I/O worker #1711] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.786 [DEBUG - New I/O worker #1710] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.785 [DEBUG - New I/O worker #1743] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.785 [DEBUG - New I/O worker #1746] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.785 [DEBUG - New I/O worker #1745] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.781 [DEBUG - New I/O worker #1686] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.780 [DEBUG - New I/O worker #2033] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:39.778 [DEBUG - New I/O worker #1685] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.778 [DEBUG - New I/O worker #1826] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:39.778 [DEBUG - New I/O worker #1687] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.777 [DEBUG - New I/O worker #2034] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:39.774 [DEBUG - New I/O worker #1729] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:39.918 [INFO - New I/O worker #1729] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:39.774 [DEBUG - New I/O worker #1847] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:39.919 [INFO - New I/O worker #1847] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:39.774 [DEBUG - pool-3-thread-1] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:39.774 [DEBUG - New I/O worker #1730] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:39.774 [DEBUG - New I/O worker #2051] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:39.919 [INFO - New I/O worker #2051] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:39.774 [DEBUG - New I/O worker #1754] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:39.773 [DEBUG - pool-3-thread-17] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:39.773 [DEBUG - pool-3-thread-33] (Deferred.java:1330) callback=disconnect callback@905821673 returned Deferred@610281010(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@636743313 after disconnect callback@905821673), errback=passthrough -> (continuation of Deferred@636743313 after disconnect callback@905821673)), so the following Deferred is getting paused: Deferred@636743313(state=PAUSED, result=Deferred@610281010, callback=, errback=) 00:27:39.773 [DEBUG - pool-3-thread-11] (Deferred.java:1330) callback=disconnect callback@889321077 returned Deferred@1080943837(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1803285100 after disconnect callback@889321077), errback=passthrough -> (continuation of Deferred@1803285100 after disconnect callback@889321077)), so the following Deferred is getting paused: Deferred@1803285100(state=PAUSED, result=Deferred@1080943837, callback=, errback=) 00:27:39.773 [DEBUG - pool-3-thread-9] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:39.758 [DEBUG - New I/O worker #1677] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:39.758 [DEBUG - New I/O worker #1877] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.757 [DEBUG - New I/O worker #1828] (Negotiator.java:764) Authenticated connection [id: 0x2917d92b, /127.0.0.1:53742 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.756 [DEBUG - New I/O worker #1907] (Negotiator.java:764) Authenticated connection [id: 0x2d800af5, /127.0.0.1:40420 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.754 [DEBUG - New I/O worker #1862] (Negotiator.java:764) Authenticated connection [id: 0x28ea6038, /127.0.0.1:40412 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.752 [DEBUG - New I/O worker #1779] (Negotiator.java:764) Authenticated connection [id: 0xd1bad697, /127.0.0.1:40390 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.752 [DEBUG - New I/O worker #1769] (Negotiator.java:764) Authenticated connection [id: 0xba3efd05, /127.0.0.1:38420 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.751 [DEBUG - New I/O worker #1778] (Negotiator.java:764) Authenticated connection [id: 0x84d1f41b, /127.0.0.1:53760 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.750 [DEBUG - New I/O worker #1777] (Negotiator.java:764) Authenticated connection [id: 0x37154602, /127.0.0.1:38414 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.748 [DEBUG - New I/O worker #1771] (Negotiator.java:764) Authenticated connection [id: 0x2b8aed7d, /127.0.0.1:40414 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.747 [INFO - New I/O worker #1753] (Connection.java:435) [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:39.925 [DEBUG - New I/O worker #1753] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:39.925 [INFO - New I/O worker #1753] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:39.747 [DEBUG - New I/O worker #1786] (Negotiator.java:764) Authenticated connection [id: 0x9cefc8e0, /127.0.0.1:38408 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.747 [DEBUG - New I/O worker #1827] (Negotiator.java:764) Authenticated connection [id: 0x0c77ee45, /127.0.0.1:38396 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.738 [DEBUG - New I/O worker #1835] (Negotiator.java:764) Authenticated connection [id: 0x2b32da19, /127.0.0.1:38390 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.737 [DEBUG - New I/O worker #1789] (Negotiator.java:764) Authenticated connection [id: 0x5196710e, /127.0.0.1:40384 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.732 [DEBUG - New I/O worker #1860] (Negotiator.java:764) Authenticated connection [id: 0xd0199e0c, /127.0.0.1:38436 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.930 [DEBUG - New I/O worker #1835] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.930 [DEBUG - New I/O worker #1835] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=667), Traces: [0ms] refreshing cache from master, [161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [389ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [478ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [666ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:39.930 [DEBUG - New I/O worker #1835] (Deferred.java:1330) callback=retry RPC@358657533 returned Deferred@291453833(state=PENDING, result=null, callback=(continuation of Deferred@1415423942 after retry RPC@358657533), errback=(continuation of Deferred@1415423942 after retry RPC@358657533)), so the following Deferred is getting paused: Deferred@1415423942(state=PAUSED, result=Deferred@291453833, callback=org.apache.kudu.client.AsyncKuduClient$6@755c052c -> wakeup thread pool-3-thread-38, errback=passthrough -> wakeup thread pool-3-thread-38) 00:27:39.933 [DEBUG - New I/O worker #1835] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.934 [DEBUG - pool-3-thread-2] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.937 [DEBUG - New I/O worker #1835] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:39.937 [DEBUG - New I/O worker #1835] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:39.937 [DEBUG - New I/O worker #1835] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:39.938 [DEBUG - New I/O worker #1835] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:39.938 [DEBUG - New I/O worker #1835] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:39.940 [DEBUG - New I/O worker #1837] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:39.940 [DEBUG - New I/O worker #1835] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:39.942 [DEBUG - New I/O worker #1935] (Negotiator.java:764) Authenticated connection [id: 0x0e58af9c, /127.0.0.1:40464 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.944 [DEBUG - New I/O worker #1912] (Negotiator.java:764) Authenticated connection [id: 0x45fa22dd, /127.0.0.1:38476 => /127.22.19.252:32833] using SASL/PLAIN 00:27:39.948 [DEBUG - pool-3-thread-36] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.950 [DEBUG - pool-3-thread-76] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.950 [DEBUG - pool-3-thread-29] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:39.912 [DEBUG - New I/O worker #1904] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:39.899 [DEBUG - New I/O worker #1861] (Negotiator.java:764) Authenticated connection [id: 0xbe41c617, /127.0.0.1:53782 => /127.22.19.254:44849] using SASL/PLAIN 00:27:39.963 [DEBUG - New I/O worker #1827] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:39.963 [DEBUG - New I/O worker #1827] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=680), Traces: [0ms] refreshing cache from master, [143ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [143ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [143ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [622ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [655ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [672ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:39.964 [DEBUG - New I/O worker #1827] (Deferred.java:1330) callback=retry RPC@1507458257 returned Deferred@1575383858(state=PENDING, result=null, callback=(continuation of Deferred@417658602 after retry RPC@1507458257), errback=(continuation of Deferred@417658602 after retry RPC@1507458257)), so the following Deferred is getting paused: Deferred@417658602(state=PAUSED, result=Deferred@1575383858, callback=org.apache.kudu.client.AsyncKuduClient$6@3f1f21bc -> wakeup thread pool-3-thread-31, errback=passthrough -> wakeup thread pool-3-thread-31) 00:27:39.992 [DEBUG - New I/O worker #2034] (Negotiator.java:764) Authenticated connection [id: 0x8a34ad93, /127.0.0.1:40434 => /127.22.19.253:37879] using SASL/PLAIN 00:27:39.999 [DEBUG - New I/O worker #1712] (Negotiator.java:764) Authenticated connection [id: 0x30cf2841, /127.0.0.1:40444 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.001 [DEBUG - New I/O worker #1746] (Negotiator.java:764) Authenticated connection [id: 0xc0e83a59, /127.0.0.1:40446 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.005 [DEBUG - New I/O worker #1942] (Negotiator.java:764) Authenticated connection [id: 0x7a631162, /127.0.0.1:53822 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.009 [DEBUG - pool-3-thread-67] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.009 [DEBUG - New I/O worker #1769] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.009 [DEBUG - New I/O worker #1769] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=669), Traces: [0ms] refreshing cache from master, [90ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [91ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [107ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [294ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [605ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [669ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.010 [DEBUG - New I/O worker #1769] (Deferred.java:1330) callback=retry RPC@169116662 returned Deferred@237700716(state=PENDING, result=null, callback=(continuation of Deferred@1261013453 after retry RPC@169116662), errback=(continuation of Deferred@1261013453 after retry RPC@169116662)), so the following Deferred is getting paused: Deferred@1261013453(state=PAUSED, result=Deferred@237700716, callback=org.apache.kudu.client.AsyncKuduClient$6@548db639 -> wakeup thread pool-3-thread-49, errback=passthrough -> wakeup thread pool-3-thread-49) 00:27:40.011 [DEBUG - New I/O worker #1686] (Negotiator.java:764) Authenticated connection [id: 0x5c6fe6e0, /127.0.0.1:53800 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.012 [DEBUG - New I/O worker #1745] (Negotiator.java:764) Authenticated connection [id: 0x60e567f1, /127.0.0.1:53814 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.018 [DEBUG - New I/O worker #1943] (Negotiator.java:764) Authenticated connection [id: 0x6d1aa07a, /127.0.0.1:40458 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.018 [DEBUG - New I/O worker #1786] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.018 [DEBUG - New I/O worker #1786] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=692), Traces: [0ms] refreshing cache from master, [102ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [102ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [102ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [347ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [624ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [692ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.019 [DEBUG - New I/O worker #1769] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.019 [DEBUG - New I/O worker #1786] (Deferred.java:1330) callback=retry RPC@982003527 returned Deferred@1052228814(state=PENDING, result=null, callback=(continuation of Deferred@2075469180 after retry RPC@982003527), errback=(continuation of Deferred@2075469180 after retry RPC@982003527)), so the following Deferred is getting paused: Deferred@2075469180(state=PAUSED, result=Deferred@1052228814, callback=org.apache.kudu.client.AsyncKuduClient$6@310def93 -> wakeup thread pool-3-thread-22, errback=passthrough -> wakeup thread pool-3-thread-22) 00:27:40.020 [DEBUG - New I/O worker #1826] (Negotiator.java:764) Authenticated connection [id: 0xdc4417b8, /127.0.0.1:38450 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.021 [DEBUG - New I/O worker #1719] (Negotiator.java:764) Authenticated connection [id: 0x6a267c4e, /127.0.0.1:53826 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.022 [DEBUG - New I/O worker #1786] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.022 [DEBUG - New I/O worker #1827] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.024 [DEBUG - New I/O worker #1777] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.024 [DEBUG - New I/O worker #1777] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=687), Traces: [0ms] refreshing cache from master, [93ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [93ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [93ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [598ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [610ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [687ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.025 [DEBUG - New I/O worker #1777] (Deferred.java:1330) callback=retry RPC@2001784227 returned Deferred@1936676753(state=PENDING, result=null, callback=(continuation of Deferred@913164184 after retry RPC@2001784227), errback=(continuation of Deferred@913164184 after retry RPC@2001784227)), so the following Deferred is getting paused: Deferred@913164184(state=PAUSED, result=Deferred@1936676753, callback=org.apache.kudu.client.AsyncKuduClient$6@76dd9580 -> wakeup thread pool-3-thread-8, errback=passthrough -> wakeup thread pool-3-thread-8) 00:27:40.025 [DEBUG - New I/O worker #1912] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.026 [DEBUG - New I/O worker #1912] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=530), Traces: [0ms] refreshing cache from master, [295ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [296ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [300ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [525ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [530ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.030 [DEBUG - New I/O worker #1769] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.030 [DEBUG - New I/O worker #1769] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.031 [DEBUG - New I/O worker #1769] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.031 [DEBUG - New I/O worker #1769] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.031 [DEBUG - New I/O worker #1769] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.032 [DEBUG - pool-3-thread-49] (Deferred.java:1330) callback=disconnect callback@69517554 returned Deferred@2059007379(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1199196710 after disconnect callback@69517554), errback=passthrough -> (continuation of Deferred@1199196710 after disconnect callback@69517554)), so the following Deferred is getting paused: Deferred@1199196710(state=PAUSED, result=Deferred@2059007379, callback=, errback=) 00:27:40.032 [DEBUG - New I/O worker #1770] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.032 [DEBUG - New I/O worker #1771] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.033 [DEBUG - New I/O worker #1710] (Negotiator.java:764) Authenticated connection [id: 0x7bae572d, /127.0.0.1:38464 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.033 [DEBUG - New I/O worker #1769] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.035 [DEBUG - New I/O worker #1786] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.035 [DEBUG - New I/O worker #1786] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.036 [DEBUG - New I/O worker #1786] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.036 [DEBUG - New I/O worker #1786] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.036 [DEBUG - New I/O worker #1786] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.037 [DEBUG - pool-3-thread-22] (Deferred.java:1330) callback=disconnect callback@702477251 returned Deferred@993977606(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@547330241 after disconnect callback@702477251), errback=passthrough -> (continuation of Deferred@547330241 after disconnect callback@702477251)), so the following Deferred is getting paused: Deferred@547330241(state=PAUSED, result=Deferred@993977606, callback=, errback=) 00:27:40.037 [DEBUG - New I/O worker #1788] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.037 [DEBUG - New I/O worker #1827] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.037 [DEBUG - New I/O worker #1827] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.038 [DEBUG - New I/O worker #1827] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.038 [DEBUG - New I/O worker #1827] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.038 [DEBUG - New I/O worker #1827] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.039 [DEBUG - New I/O worker #1777] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.039 [DEBUG - New I/O worker #1828] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.039 [DEBUG - pool-3-thread-31] (Deferred.java:1330) callback=disconnect callback@1597859932 returned Deferred@1735419859(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1402129518 after disconnect callback@1597859932), errback=passthrough -> (continuation of Deferred@1402129518 after disconnect callback@1597859932)), so the following Deferred is getting paused: Deferred@1402129518(state=PAUSED, result=Deferred@1735419859, callback=, errback=) 00:27:40.040 [DEBUG - New I/O worker #1829] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.040 [DEBUG - New I/O worker #1827] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.043 [DEBUG - New I/O worker #1777] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.043 [DEBUG - New I/O worker #1777] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.044 [DEBUG - New I/O worker #1777] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.044 [DEBUG - New I/O worker #1777] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.044 [DEBUG - New I/O worker #1777] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.045 [DEBUG - pool-3-thread-8] (Deferred.java:1330) callback=disconnect callback@1456462405 returned Deferred@400167520(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1750717023 after disconnect callback@1456462405), errback=passthrough -> (continuation of Deferred@1750717023 after disconnect callback@1456462405)), so the following Deferred is getting paused: Deferred@1750717023(state=PAUSED, result=Deferred@400167520, callback=, errback=) 00:27:40.045 [DEBUG - New I/O worker #1778] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.048 [DEBUG - New I/O worker #1914] (Negotiator.java:764) Authenticated connection [id: 0x3cde7aeb, /127.0.0.1:38488 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.048 [DEBUG - New I/O worker #1777] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.049 [DEBUG - New I/O worker #1710] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.049 [DEBUG - New I/O worker #1710] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=537), Traces: [0ms] refreshing cache from master, [269ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [269ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [269ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [493ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [537ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.031 [DEBUG - New I/O worker #1826] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.050 [DEBUG - New I/O worker #1743] (Negotiator.java:764) Authenticated connection [id: 0xc9733830, /127.0.0.1:38466 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.051 [DEBUG - New I/O worker #1718] (Negotiator.java:764) Authenticated connection [id: 0x8d2c4b3f, /127.0.0.1:38480 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.051 [DEBUG - New I/O worker #1710] (Deferred.java:1330) callback=retry RPC@1939390304 returned Deferred@2007446987(state=PENDING, result=null, callback=(continuation of Deferred@849720667 after retry RPC@1939390304), errback=(continuation of Deferred@849720667 after retry RPC@1939390304)), so the following Deferred is getting paused: Deferred@849720667(state=PAUSED, result=Deferred@2007446987, callback=org.apache.kudu.client.AsyncKuduClient$6@35089f32 -> wakeup thread pool-3-thread-41, errback=passthrough -> wakeup thread pool-3-thread-41) 00:27:40.052 [DEBUG - New I/O worker #1710] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.053 [DEBUG - New I/O worker #1914] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.053 [DEBUG - New I/O worker #1914] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=421), Traces: [0ms] refreshing cache from master, [127ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [259ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [259ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [319ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [421ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.053 [DEBUG - New I/O worker #1914] (Deferred.java:1330) callback=retry RPC@1444067085 returned Deferred@1378691351(state=PENDING, result=null, callback=(continuation of Deferred@389004598 after retry RPC@1444067085), errback=(continuation of Deferred@389004598 after retry RPC@1444067085)), so the following Deferred is getting paused: Deferred@389004598(state=PAUSED, result=Deferred@1378691351, callback=org.apache.kudu.client.AsyncKuduClient$6@651b13ce -> wakeup thread pool-3-thread-65, errback=passthrough -> wakeup thread pool-3-thread-65) 00:27:40.054 [DEBUG - New I/O worker #1718] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.054 [DEBUG - New I/O worker #1718] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=601), Traces: [0ms] refreshing cache from master, [340ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [340ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [341ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [573ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [602ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.055 [DEBUG - New I/O worker #1860] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.055 [DEBUG - New I/O worker #1860] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=796), Traces: [0ms] refreshing cache from master, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [685ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [695ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [796ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.055 [DEBUG - New I/O worker #1743] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.056 [DEBUG - New I/O worker #1743] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=551), Traces: [0ms] refreshing cache from master, [278ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [278ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [278ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [502ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [514ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [552ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.056 [DEBUG - New I/O worker #1743] (Deferred.java:1330) callback=retry RPC@2018049311 returned Deferred@2088199886(state=PENDING, result=null, callback=(continuation of Deferred@964032292 after retry RPC@2018049311), errback=(continuation of Deferred@964032292 after retry RPC@2018049311)), so the following Deferred is getting paused: Deferred@964032292(state=PAUSED, result=Deferred@2088199886, callback=org.apache.kudu.client.AsyncKuduClient$6@3799ff49 -> wakeup thread pool-3-thread-16, errback=passthrough -> wakeup thread pool-3-thread-16) 00:27:40.049 [DEBUG - New I/O worker #1934] (Negotiator.java:764) Authenticated connection [id: 0x847bdeca, /127.0.0.1:53834 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.047 [DEBUG - New I/O worker #1779] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.058 [DEBUG - New I/O worker #1718] (Deferred.java:1330) callback=retry RPC@86309192 returned Deferred@18574644(state=PENDING, result=null, callback=(continuation of Deferred@1142551411 after retry RPC@86309192), errback=(continuation of Deferred@1142551411 after retry RPC@86309192)), so the following Deferred is getting paused: Deferred@1142551411(state=PAUSED, result=Deferred@18574644, callback=org.apache.kudu.client.AsyncKuduClient$6@57db1c92 -> wakeup thread pool-3-thread-7, errback=passthrough -> wakeup thread pool-3-thread-7) 00:27:40.058 [DEBUG - New I/O worker #1718] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.047 [DEBUG - New I/O worker #1720] (Negotiator.java:764) Authenticated connection [id: 0xd6602452, /127.0.0.1:40456 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.047 [DEBUG - New I/O worker #1687] (Negotiator.java:764) Authenticated connection [id: 0x29316d84, /127.0.0.1:40432 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.060 [DEBUG - New I/O worker #1914] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.043 [DEBUG - New I/O worker #1904] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.061 [DEBUG - New I/O worker #1904] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.061 [DEBUG - New I/O worker #1904] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.061 [DEBUG - New I/O worker #1904] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.061 [DEBUG - New I/O worker #1904] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.062 [DEBUG - pool-3-thread-21] (Deferred.java:1330) callback=disconnect callback@667344263 returned Deferred@1214061793(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@457043672 after disconnect callback@667344263), errback=passthrough -> (continuation of Deferred@457043672 after disconnect callback@667344263)), so the following Deferred is getting paused: Deferred@457043672(state=PAUSED, result=Deferred@1214061793, callback=, errback=) 00:27:40.062 [DEBUG - New I/O worker #1907] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.063 [DEBUG - New I/O worker #1906] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.063 [DEBUG - New I/O worker #1743] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.064 [DEBUG - New I/O worker #1710] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.064 [DEBUG - New I/O worker #1710] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.064 [DEBUG - New I/O worker #1710] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.064 [DEBUG - New I/O worker #1710] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.065 [DEBUG - New I/O worker #1710] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.065 [DEBUG - pool-3-thread-41] (Deferred.java:1330) callback=disconnect callback@1010147123 returned Deferred@1656266861(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1090280429 after disconnect callback@1010147123), errback=passthrough -> (continuation of Deferred@1090280429 after disconnect callback@1010147123)), so the following Deferred is getting paused: Deferred@1090280429(state=PAUSED, result=Deferred@1656266861, callback=, errback=) 00:27:40.066 [DEBUG - New I/O worker #1712] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.066 [DEBUG - New I/O worker #1904] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.067 [DEBUG - New I/O worker #1743] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.067 [DEBUG - New I/O worker #1743] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.067 [DEBUG - New I/O worker #1743] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.067 [DEBUG - New I/O worker #1743] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.068 [DEBUG - New I/O worker #1743] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.068 [DEBUG - pool-3-thread-34] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.068 [DEBUG - New I/O worker #1685] (Negotiator.java:764) Authenticated connection [id: 0x0fc40f5f, /127.0.0.1:38452 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.069 [DEBUG - New I/O worker #1746] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.069 [DEBUG - pool-3-thread-32] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.069 [DEBUG - New I/O worker #1743] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.043 [DEBUG - New I/O worker #2033] (Negotiator.java:764) Authenticated connection [id: 0x4660686b, /127.0.0.1:53802 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.070 [DEBUG - pool-3-thread-35] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.072 [DEBUG - New I/O worker #1685] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.072 [DEBUG - New I/O worker #1685] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=562), Traces: [0ms] refreshing cache from master, [264ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [264ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [265ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [504ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [559ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [562ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.074 [DEBUG - New I/O worker #1685] (Deferred.java:1330) callback=retry RPC@619458534 returned Deferred@550743971(state=PENDING, result=null, callback=(continuation of Deferred@1708205533 after retry RPC@619458534), errback=(continuation of Deferred@1708205533 after retry RPC@619458534)), so the following Deferred is getting paused: Deferred@1708205533(state=PAUSED, result=Deferred@550743971, callback=org.apache.kudu.client.AsyncKuduClient$6@112878e8 -> wakeup thread pool-3-thread-50, errback=passthrough -> wakeup thread pool-3-thread-50) 00:27:40.039 [DEBUG - New I/O worker #1711] (Negotiator.java:764) Authenticated connection [id: 0x9eed4388, /127.0.0.1:53812 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.038 [DEBUG - New I/O worker #1786] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.037 [DEBUG - New I/O worker #1789] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.076 [DEBUG - New I/O worker #1711] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.031 [DEBUG - New I/O worker #1912] (Deferred.java:1330) callback=retry RPC@633658739 returned Deferred@570117395(state=PENDING, result=null, callback=(continuation of Deferred@1694095176 after retry RPC@633658739), errback=(continuation of Deferred@1694095176 after retry RPC@633658739)), so the following Deferred is getting paused: Deferred@1694095176(state=PAUSED, result=Deferred@570117395, callback=org.apache.kudu.client.AsyncKuduClient$6@56fbc58a -> wakeup thread pool-3-thread-61, errback=passthrough -> wakeup thread pool-3-thread-61) 00:27:40.068 [DEBUG - pool-3-thread-16] (Deferred.java:1330) callback=disconnect callback@343221714 returned Deferred@1470345060(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1844806829 after disconnect callback@343221714), errback=passthrough -> (continuation of Deferred@1844806829 after disconnect callback@343221714)), so the following Deferred is getting paused: Deferred@1844806829(state=PAUSED, result=Deferred@1470345060, callback=, errback=) 00:27:40.077 [DEBUG - New I/O worker #1912] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.077 [DEBUG - New I/O worker #1914] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.077 [DEBUG - New I/O worker #1914] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.077 [DEBUG - New I/O worker #1718] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.078 [DEBUG - New I/O worker #1718] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.078 [DEBUG - New I/O worker #1718] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.078 [DEBUG - New I/O worker #1718] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.078 [DEBUG - New I/O worker #1745] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.067 [DEBUG - New I/O worker #1710] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.059 [DEBUG - pool-3-thread-10] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.055 [DEBUG - New I/O worker #1860] (Deferred.java:1330) callback=retry RPC@313494773 returned Deferred@378542712(state=PENDING, result=null, callback=(continuation of Deferred@1402111694 after retry RPC@313494773), errback=(continuation of Deferred@1402111694 after retry RPC@313494773)), so the following Deferred is getting paused: Deferred@1402111694(state=PAUSED, result=Deferred@378542712, callback=org.apache.kudu.client.AsyncKuduClient$6@c2d4d85 -> wakeup thread pool-3-thread-66, errback=passthrough -> wakeup thread pool-3-thread-66) 00:27:40.050 [DEBUG - New I/O worker #1826] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=547), Traces: [0ms] refreshing cache from master, [271ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [271ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [272ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [491ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [528ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.079 [DEBUG - New I/O worker #1826] (Deferred.java:1330) callback=retry RPC@331657785 returned Deferred@402334873(state=PENDING, result=null, callback=(continuation of Deferred@1392097282 after retry RPC@331657785), errback=(continuation of Deferred@1392097282 after retry RPC@331657785)), so the following Deferred is getting paused: Deferred@1392097282(state=PAUSED, result=Deferred@402334873, callback=org.apache.kudu.client.AsyncKuduClient$6@5582d9b3 -> wakeup thread pool-3-thread-47, errback=passthrough -> wakeup thread pool-3-thread-47) 00:27:40.078 [DEBUG - New I/O worker #1718] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.077 [DEBUG - New I/O worker #1914] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.080 [DEBUG - New I/O worker #1914] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.080 [DEBUG - pool-3-thread-7] (Deferred.java:1330) callback=disconnect callback@2112730743 returned Deferred@222776814(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1342275084 after disconnect callback@2112730743), errback=passthrough -> (continuation of Deferred@1342275084 after disconnect callback@2112730743)), so the following Deferred is getting paused: Deferred@1342275084(state=PAUSED, result=Deferred@222776814, callback=, errback=) 00:27:40.080 [DEBUG - New I/O worker #1914] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.081 [DEBUG - New I/O worker #1719] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.081 [DEBUG - pool-3-thread-65] (Deferred.java:1330) callback=disconnect callback@605552069 returned Deferred@422238181(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1882521068 after disconnect callback@605552069), errback=passthrough -> (continuation of Deferred@1882521068 after disconnect callback@605552069)), so the following Deferred is getting paused: Deferred@1882521068(state=PAUSED, result=Deferred@422238181, callback=, errback=) 00:27:40.082 [DEBUG - New I/O worker #1935] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.082 [DEBUG - New I/O worker #1914] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.082 [DEBUG - New I/O worker #1718] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.082 [DEBUG - New I/O worker #1720] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.086 [DEBUG - New I/O worker #1685] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.087 [DEBUG - New I/O worker #1934] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.089 [DEBUG - pool-3-thread-55] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.089 [DEBUG - New I/O worker #1777] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.090 [DEBUG - New I/O worker #1912] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.090 [DEBUG - New I/O worker #1826] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.090 [DEBUG - New I/O worker #1827] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.090 [DEBUG - New I/O worker #1912] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.090 [DEBUG - New I/O worker #1685] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.091 [DEBUG - New I/O worker #1912] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.091 [DEBUG - New I/O worker #1912] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.091 [DEBUG - New I/O worker #1912] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.092 [DEBUG - pool-3-thread-61] (Deferred.java:1330) callback=disconnect callback@54233688 returned Deferred@1038876892(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1369218848 after disconnect callback@54233688), errback=passthrough -> (continuation of Deferred@1369218848 after disconnect callback@54233688)), so the following Deferred is getting paused: Deferred@1369218848(state=PAUSED, result=Deferred@1038876892, callback=, errback=) 00:27:40.092 [DEBUG - pool-3-thread-78] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.209 [DEBUG - New I/O worker #1951] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.213 [DEBUG - New I/O worker #1942] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.213 [DEBUG - New I/O worker #1943] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.209 [DEBUG - New I/O worker #1860] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.208 [DEBUG - New I/O worker #1685] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.217 [DEBUG - New I/O worker #1685] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.217 [DEBUG - New I/O worker #1685] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.217 [DEBUG - New I/O worker #1685] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.218 [DEBUG - pool-3-thread-50] (Deferred.java:1330) callback=disconnect callback@1006228643 returned Deferred@637861642(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@338659706 after disconnect callback@1006228643), errback=passthrough -> (continuation of Deferred@338659706 after disconnect callback@1006228643)), so the following Deferred is getting paused: Deferred@338659706(state=PAUSED, result=Deferred@637861642, callback=, errback=) 00:27:40.218 [DEBUG - New I/O worker #1686] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.219 [DEBUG - New I/O worker #1687] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.208 [DEBUG - New I/O worker #1912] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.208 [DEBUG - pool-3-thread-80] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.208 [DEBUG - New I/O worker #1997] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.220 [DEBUG - New I/O worker #1987] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.220 [DEBUG - New I/O worker #1685] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.207 [DEBUG - New I/O worker #2027] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.205 [DEBUG - New I/O worker #1958] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.204 [DEBUG - New I/O worker #1986] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.203 [DEBUG - New I/O worker #1804] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.221 [DEBUG - New I/O worker #1905] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.203 [DEBUG - pool-3-thread-64] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.200 [DEBUG - New I/O worker #1898] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.222 [DEBUG - pool-3-thread-73] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.222 [DEBUG - New I/O worker #1934] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.198 [DEBUG - New I/O worker #1707] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.194 [DEBUG - New I/O worker #1903] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.194 [DEBUG - New I/O worker #2022] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.190 [DEBUG - New I/O worker #1901] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.188 [DEBUG - New I/O worker #1868] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.188 [DEBUG - New I/O worker #1885] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.186 [DEBUG - New I/O worker #1900] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.184 [DEBUG - New I/O worker #1957] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.183 [DEBUG - New I/O worker #1875] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.182 [DEBUG - New I/O worker #2021] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.180 [DEBUG - New I/O worker #2148] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.141 [DEBUG - New I/O worker #1817] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.135 [DEBUG - New I/O worker #1998] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.134 [DEBUG - New I/O worker #1881] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.133 [DEBUG - New I/O worker #1879] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.133 [DEBUG - New I/O worker #2041] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.132 [DEBUG - New I/O worker #1819] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.130 [DEBUG - New I/O worker #2042] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.130 [DEBUG - New I/O worker #1761] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.129 [DEBUG - New I/O worker #1850] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.127 [DEBUG - New I/O worker #2005] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.127 [DEBUG - New I/O worker #1763] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.127 [DEBUG - New I/O worker #2004] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.126 [DEBUG - New I/O worker #1762] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.125 [DEBUG - New I/O worker #1820] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.124 [DEBUG - New I/O worker #1821] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.122 [DEBUG - New I/O worker #1728] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.121 [DEBUG - New I/O worker #2075] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.121 [DEBUG - New I/O worker #1738] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.120 [DEBUG - New I/O worker #2076] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.119 [DEBUG - New I/O worker #2129] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.118 [DEBUG - New I/O worker #2072] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.118 [DEBUG - pool-3-thread-60] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.117 [DEBUG - pool-3-thread-63] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.116 [DEBUG - pool-3-thread-19] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.116 [DEBUG - New I/O worker #1730] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.115 [DEBUG - New I/O worker #1806] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.115 [DEBUG - New I/O worker #1754] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.115 [DEBUG - New I/O worker #2060] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.115 [DEBUG - New I/O worker #2058] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.114 [DEBUG - New I/O worker #1857] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.113 [DEBUG - New I/O worker #1856] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.113 [DEBUG - New I/O worker #1854] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.107 [DEBUG - New I/O worker #1838] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.107 [DEBUG - pool-3-thread-38] (Deferred.java:1330) callback=disconnect callback@108380231 returned Deferred@695843246(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1196903468 after disconnect callback@108380231), errback=passthrough -> (continuation of Deferred@1196903468 after disconnect callback@108380231)), so the following Deferred is getting paused: Deferred@1196903468(state=PAUSED, result=Deferred@695843246, callback=, errback=) 00:27:40.238 [DEBUG - New I/O worker #2061] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.220 [DEBUG - New I/O worker #1826] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.243 [DEBUG - New I/O worker #1826] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.272 [DEBUG - New I/O worker #1826] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.272 [DEBUG - New I/O worker #1826] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.215 [DEBUG - New I/O worker #2083] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.215 [DEBUG - New I/O worker #2028] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.215 [DEBUG - New I/O worker #1966] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.214 [DEBUG - New I/O worker #1769] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.214 [DEBUG - New I/O worker #2023] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.277 [DEBUG - New I/O worker #1826] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.288 [DEBUG - pool-3-thread-47] (Deferred.java:1330) callback=disconnect callback@2088367465 returned Deferred@126577255(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1204501612 after disconnect callback@2088367465), errback=passthrough -> (continuation of Deferred@1204501612 after disconnect callback@2088367465)), so the following Deferred is getting paused: Deferred@1204501612(state=PAUSED, result=Deferred@126577255, callback=, errback=) 00:27:40.289 [DEBUG - New I/O worker #2033] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.289 [DEBUG - New I/O worker #2034] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.272 [DEBUG - New I/O worker #1786] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.271 [DEBUG - New I/O worker #1711] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.292 [INFO - New I/O worker #1711] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:40.271 [DEBUG - New I/O worker #1745] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.271 [DEBUG - New I/O worker #1718] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.295 [DEBUG - New I/O worker #1965] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.296 [DEBUG - New I/O worker #1826] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.268 [DEBUG - New I/O worker #1949] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.268 [DEBUG - New I/O worker #1819] (Negotiator.java:764) Authenticated connection [id: 0x79f881b7, /127.0.0.1:38520 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.266 [DEBUG - New I/O worker #2027] (Negotiator.java:764) Authenticated connection [id: 0x5e1c8974, /127.0.0.1:53876 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.304 [DEBUG - New I/O worker #2062] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.261 [DEBUG - New I/O worker #1903] (Negotiator.java:764) Authenticated connection [id: 0x7dafe60e, /127.0.0.1:53928 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.249 [DEBUG - New I/O worker #1984] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.289 [DEBUG - New I/O worker #1998] (Negotiator.java:764) Authenticated connection [id: 0xddfffc34, /127.0.0.1:40520 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.306 [DEBUG - New I/O worker #1819] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.308 [DEBUG - New I/O worker #1819] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=396), Traces: [0ms] refreshing cache from master, [211ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [212ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [212ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [396ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.310 [DEBUG - New I/O worker #2163] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.313 [DEBUG - New I/O worker #2041] (Negotiator.java:764) Authenticated connection [id: 0xc63f4c05, /127.0.0.1:53890 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.314 [DEBUG - New I/O worker #1709] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.319 [DEBUG - New I/O worker #1997] (Negotiator.java:764) Authenticated connection [id: 0xf9501a59, /127.0.0.1:53888 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.319 [DEBUG - New I/O worker #1761] (Negotiator.java:764) Authenticated connection [id: 0xdbd9d094, /127.0.0.1:38518 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.320 [DEBUG - New I/O worker #2096] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.322 [DEBUG - New I/O worker #2004] (Negotiator.java:764) Authenticated connection [id: 0x20e0d89a, /127.0.0.1:53880 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.322 [DEBUG - New I/O worker #1761] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.323 [DEBUG - New I/O worker #1860] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.324 [DEBUG - New I/O worker #1860] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.324 [DEBUG - New I/O worker #1860] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.324 [DEBUG - New I/O worker #1860] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.324 [DEBUG - New I/O worker #1860] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.327 [DEBUG - New I/O worker #1861] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.327 [DEBUG - New I/O worker #1862] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.328 [DEBUG - pool-3-thread-66] (Deferred.java:1330) callback=disconnect callback@696652973 returned Deferred@478345713(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1447630872 after disconnect callback@696652973), errback=passthrough -> (continuation of Deferred@1447630872 after disconnect callback@696652973)), so the following Deferred is getting paused: Deferred@1447630872(state=PAUSED, result=Deferred@478345713, callback=, errback=) 00:27:40.323 [DEBUG - New I/O worker #1892] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.327 [DEBUG - New I/O worker #1904] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.329 [DEBUG - New I/O worker #1860] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.329 [DEBUG - New I/O worker #1761] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=418), Traces: [0ms] refreshing cache from master, [210ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [210ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [211ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [411ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.332 [DEBUG - New I/O worker #1860] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.334 [DEBUG - New I/O worker #1826] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.334 [DEBUG - New I/O worker #2072] (Negotiator.java:764) Authenticated connection [id: 0xfc4d570f, /127.0.0.1:53852 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.335 [DEBUG - New I/O worker #1838] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.337 [DEBUG - New I/O worker #1821] (Negotiator.java:764) Authenticated connection [id: 0x7bcdbe5b, /127.0.0.1:40500 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.339 [DEBUG - New I/O worker #1763] (Negotiator.java:764) Authenticated connection [id: 0x76155641, /127.0.0.1:40498 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.341 [DEBUG - New I/O worker #1856] (Negotiator.java:764) Authenticated connection [id: 0x4baec043, /127.0.0.1:53840 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.373 [DEBUG - New I/O worker #1819] (Deferred.java:1330) callback=retry RPC@704081765 returned Deferred@768142979(state=PENDING, result=null, callback=(continuation of Deferred@1758094686 after retry RPC@704081765), errback=(continuation of Deferred@1758094686 after retry RPC@704081765)), so the following Deferred is getting paused: Deferred@1758094686(state=PAUSED, result=Deferred@768142979, callback=org.apache.kudu.client.AsyncKuduClient$6@4bf66aa9 -> wakeup thread pool-3-thread-75, errback=passthrough -> wakeup thread pool-3-thread-75) 00:27:40.377 [DEBUG - New I/O worker #1819] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.380 [DEBUG - New I/O worker #1819] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.380 [DEBUG - New I/O worker #1819] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.381 [DEBUG - New I/O worker #1819] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.381 [DEBUG - New I/O worker #1819] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.381 [DEBUG - New I/O worker #1819] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.384 [DEBUG - New I/O worker #1879] (Negotiator.java:764) Authenticated connection [id: 0x857bc492, /127.0.0.1:38534 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:40.384833 26088 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.123518 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53868 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.123942 (+ 424us) server_negotiation.cc:184] Beginning negotiation 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.123978 (+ 36us) server_negotiation.cc:373] Waiting for connection header 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.124024 (+ 46us) server_negotiation.cc:381] Connection header received 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.124130 (+ 106us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.124156 (+ 26us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.124303 (+ 147us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.124462 (+ 159us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.291608 (+167146us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.292691 (+ 1083us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.383610 (+ 90919us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53868: BlockingRecv error: recv got EOF from 127.0.0.1:53868 (error 108) 00:27:40.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":138} 00:27:40.386 [DEBUG - New I/O worker #1819] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.388 [DEBUG - New I/O worker #2129] (Negotiator.java:764) Authenticated connection [id: 0x0f3a6993, /127.0.0.1:40482 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.385 [DEBUG - New I/O worker #2021] (Negotiator.java:764) Authenticated connection [id: 0x86089583, /127.0.0.1:53910 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.391 [DEBUG - New I/O worker #1879] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.391 [DEBUG - New I/O worker #1881] (Negotiator.java:764) Authenticated connection [id: 0xfd3e61ab, /127.0.0.1:38540 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.391 [DEBUG - New I/O worker #1738] (Negotiator.java:764) Authenticated connection [id: 0x167842dc, /127.0.0.1:38506 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.395 [DEBUG - New I/O worker #2005] (Negotiator.java:764) Authenticated connection [id: 0x6caba37f, /127.0.0.1:40510 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.398 [DEBUG - New I/O worker #2083] (Negotiator.java:764) Authenticated connection [id: 0x11045eb2, /127.0.0.1:40564 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.399 [DEBUG - New I/O worker #1881] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.399 [DEBUG - New I/O worker #1881] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=489), Traces: [0ms] refreshing cache from master, [217ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [217ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [218ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [421ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [435ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [489ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.400 [DEBUG - New I/O worker #1881] (Deferred.java:1330) callback=retry RPC@1879714849 returned Deferred@1949673208(state=PENDING, result=null, callback=(continuation of Deferred@825697818 after retry RPC@1879714849), errback=(continuation of Deferred@825697818 after retry RPC@1879714849)), so the following Deferred is getting paused: Deferred@825697818(state=PAUSED, result=Deferred@1949673208, callback=org.apache.kudu.client.AsyncKuduClient$6@21060c39 -> wakeup thread pool-3-thread-68, errback=passthrough -> wakeup thread pool-3-thread-68) 00:27:40.400 [DEBUG - New I/O worker #1738] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.400 [DEBUG - New I/O worker #1738] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=452), Traces: [0ms] refreshing cache from master, [169ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [169ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [169ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [390ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [445ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [452ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.401 [DEBUG - New I/O worker #1738] (Deferred.java:1330) callback=retry RPC@141144918 returned Deferred@206976643(state=PENDING, result=null, callback=(continuation of Deferred@1230289261 after retry RPC@141144918), errback=(continuation of Deferred@1230289261 after retry RPC@141144918)), so the following Deferred is getting paused: Deferred@1230289261(state=PAUSED, result=Deferred@206976643, callback=org.apache.kudu.client.AsyncKuduClient$6@3de13471 -> wakeup thread pool-3-thread-36, errback=passthrough -> wakeup thread pool-3-thread-36) 00:27:40.391 [DEBUG - New I/O worker #1879] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=481), Traces: [0ms] refreshing cache from master, [214ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [214ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [214ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [423ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [481ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.401 [DEBUG - New I/O worker #1879] (Deferred.java:1330) callback=retry RPC@821085063 returned Deferred@886001985(state=PENDING, result=null, callback=(continuation of Deferred@1909311932 after retry RPC@821085063), errback=(continuation of Deferred@1909311932 after retry RPC@821085063)), so the following Deferred is getting paused: Deferred@1909311932(state=PAUSED, result=Deferred@886001985, callback=org.apache.kudu.client.AsyncKuduClient$6@76ac8306 -> wakeup thread pool-3-thread-54, errback=passthrough -> wakeup thread pool-3-thread-54) 00:27:40.407 [DEBUG - New I/O worker #1738] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.408 [DEBUG - New I/O worker #1881] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.408 [DEBUG - New I/O worker #1879] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.411 [DEBUG - New I/O worker #1885] (Negotiator.java:764) Authenticated connection [id: 0x03812420, /127.0.0.1:38552 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.412 [DEBUG - New I/O worker #1881] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.412 [DEBUG - New I/O worker #1881] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.413 [DEBUG - New I/O worker #1881] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.413 [DEBUG - New I/O worker #1881] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.413 [DEBUG - New I/O worker #1881] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.414 [DEBUG - New I/O worker #1738] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.414 [DEBUG - New I/O worker #1738] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.414 [DEBUG - New I/O worker #1738] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.414 [DEBUG - New I/O worker #1738] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.414 [DEBUG - New I/O worker #1738] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.415 [DEBUG - pool-3-thread-36] (Deferred.java:1330) callback=disconnect callback@1563092621 returned Deferred@1405479290(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2144826715 after disconnect callback@1563092621), errback=passthrough -> (continuation of Deferred@2144826715 after disconnect callback@1563092621)), so the following Deferred is getting paused: Deferred@2144826715(state=PAUSED, result=Deferred@1405479290, callback=, errback=) 00:27:40.415 [DEBUG - pool-3-thread-68] (Deferred.java:1330) callback=disconnect callback@1489801997 returned Deferred@2110600694(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@474034312 after disconnect callback@1489801997), errback=passthrough -> (continuation of Deferred@474034312 after disconnect callback@1489801997)), so the following Deferred is getting paused: Deferred@474034312(state=PAUSED, result=Deferred@2110600694, callback=, errback=) 00:27:40.415 [DEBUG - New I/O worker #1997] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.416 [DEBUG - New I/O worker #1881] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.416 [DEBUG - New I/O worker #1738] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.416 [DEBUG - New I/O worker #1998] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.416 [DEBUG - New I/O worker #2129] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.417 [DEBUG - New I/O worker #2072] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.422 [DEBUG - New I/O worker #1879] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.422 [DEBUG - New I/O worker #1879] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.422 [DEBUG - New I/O worker #1879] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.423 [DEBUG - New I/O worker #1879] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.423 [DEBUG - New I/O worker #1879] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.424 [DEBUG - pool-3-thread-54] (Deferred.java:1330) callback=disconnect callback@1150312197 returned Deferred@2012032763(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1925381021 after disconnect callback@1150312197), errback=passthrough -> (continuation of Deferred@1925381021 after disconnect callback@1150312197)), so the following Deferred is getting paused: Deferred@1925381021(state=PAUSED, result=Deferred@2012032763, callback=, errback=) 00:27:40.424 [DEBUG - New I/O worker #1879] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.424 [DEBUG - New I/O worker #2005] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.425 [DEBUG - New I/O worker #2004] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.429 [DEBUG - New I/O worker #2075] (Negotiator.java:764) Authenticated connection [id: 0x039a0c71, /127.0.0.1:53858 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.434 [DEBUG - New I/O worker #1804] (Negotiator.java:764) Authenticated connection [id: 0xe27263ad, /127.0.0.1:38594 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.423 [DEBUG - New I/O worker #1728] (Negotiator.java:764) Authenticated connection [id: 0x6d59f3da, /127.0.0.1:38512 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.437 [DEBUG - New I/O worker #1857] (Negotiator.java:764) Authenticated connection [id: 0x7c5184f8, /127.0.0.1:40476 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.440 [DEBUG - New I/O worker #1709] (Negotiator.java:764) Authenticated connection [id: 0x41c18597, /127.0.0.1:53934 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.440 [DEBUG - New I/O worker #2096] (Negotiator.java:764) Authenticated connection [id: 0xb0756b56, /127.0.0.1:40584 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.426 [DEBUG - New I/O worker #1885] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.441 [DEBUG - New I/O worker #1885] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=531), Traces: [0ms] refreshing cache from master, [227ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [227ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [228ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [517ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.426 [DEBUG - New I/O worker #2062] (Negotiator.java:764) Authenticated connection [id: 0x00d9e09a, /127.0.0.1:40570 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.424 [DEBUG - New I/O worker #2061] (Negotiator.java:764) Authenticated connection [id: 0xbab5f0b7, /127.0.0.1:53940 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.515 [DEBUG - New I/O worker #1996] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.515 [DEBUG - New I/O worker #1884] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.513 [DEBUG - New I/O worker #1896] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.512 [DEBUG - New I/O worker #1882] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.509 [DEBUG - New I/O worker #1821] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.509 [DEBUG - pool-3-thread-75] (Deferred.java:1330) callback=disconnect callback@307922036 returned Deferred@448760654(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1754380282 after disconnect callback@307922036), errback=passthrough -> (continuation of Deferred@1754380282 after disconnect callback@307922036)), so the following Deferred is getting paused: Deferred@1754380282(state=PAUSED, result=Deferred@448760654, callback=, errback=) 00:27:40.467 [DEBUG - New I/O worker #1875] (Negotiator.java:764) Authenticated connection [id: 0x5325ac5f, /127.0.0.1:38564 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.466 [DEBUG - New I/O worker #1900] (Negotiator.java:764) Authenticated connection [id: 0x784e1bcc, /127.0.0.1:38570 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.465 [DEBUG - New I/O worker #1943] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.465 [DEBUG - New I/O worker #1707] (Negotiator.java:764) Authenticated connection [id: 0xc5f6ea1f, /127.0.0.1:38588 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.463 [DEBUG - New I/O worker #1868] (Negotiator.java:764) Authenticated connection [id: 0xe62f5ef0, /127.0.0.1:38576 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.460 [DEBUG - New I/O worker #1685] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.459 [DEBUG - New I/O worker #2060] (Negotiator.java:764) Authenticated connection [id: 0x08941ff0, /127.0.0.1:40474 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.459 [DEBUG - New I/O worker #2042] (Negotiator.java:764) Authenticated connection [id: 0xe37a01e6, /127.0.0.1:40522 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.455 [DEBUG - New I/O worker #1905] (Negotiator.java:764) Authenticated connection [id: 0x0e7eebf3, /127.0.0.1:38608 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.455 [DEBUG - New I/O worker #1966] (Negotiator.java:764) Authenticated connection [id: 0x4f409e2e, /127.0.0.1:40534 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.528 [DEBUG - New I/O worker #1707] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.528 [DEBUG - New I/O worker #1707] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=619), Traces: [0ms] refreshing cache from master, [280ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [492ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [535ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [619ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.528 [DEBUG - New I/O worker #1868] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.529 [DEBUG - New I/O worker #1868] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=620), Traces: [0ms] refreshing cache from master, [256ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [256ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [256ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [619ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.454 [DEBUG - New I/O worker #1984] (Negotiator.java:764) Authenticated connection [id: 0xe1234b92, /127.0.0.1:53948 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.452 [DEBUG - New I/O worker #2076] (Negotiator.java:764) Authenticated connection [id: 0x24706809, /127.0.0.1:40488 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.447 [DEBUG - New I/O worker #1761] (Deferred.java:1330) callback=retry RPC@1379047564 returned Deferred@1443698194(state=PENDING, result=null, callback=(continuation of Deferred@319787703 after retry RPC@1379047564), errback=(continuation of Deferred@319787703 after retry RPC@1379047564)), so the following Deferred is getting paused: Deferred@319787703(state=PAUSED, result=Deferred@1443698194, callback=org.apache.kudu.client.AsyncKuduClient$6@6466e3d9 -> wakeup thread pool-3-thread-52, errback=passthrough -> wakeup thread pool-3-thread-52) 00:27:40.532 [DEBUG - New I/O worker #1761] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.447 [INFO - New I/O worker #1820] (Connection.java:435) [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:40.532 [DEBUG - New I/O worker #1820] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:40.533 [INFO - New I/O worker #1820] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: [peer master-127.22.19.254:44849(127.22.19.254:44849)] disconnected from peer 00:27:40.445 [DEBUG - New I/O worker #1817] (Negotiator.java:764) Authenticated connection [id: 0x0a7b0597, /127.0.0.1:38542 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.444 [DEBUG - New I/O worker #1728] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.534 [DEBUG - New I/O worker #1728] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=623), Traces: [0ms] refreshing cache from master, [207ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [207ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [207ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [526ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [533ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.534 [DEBUG - New I/O worker #1728] (Deferred.java:1330) callback=retry RPC@1674125139 returned Deferred@1744209906(state=PENDING, result=null, callback=(continuation of Deferred@586421608 after retry RPC@1674125139), errback=(continuation of Deferred@586421608 after retry RPC@1674125139)), so the following Deferred is getting paused: Deferred@586421608(state=PAUSED, result=Deferred@1744209906, callback=org.apache.kudu.client.AsyncKuduClient$6@38849748 -> wakeup thread pool-3-thread-71, errback=passthrough -> wakeup thread pool-3-thread-71) 00:27:40.535 [DEBUG - New I/O worker #1905] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.535 [DEBUG - New I/O worker #1905] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=560), Traces: [0ms] refreshing cache from master, [240ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [240ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [240ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [471ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [560ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.535 [DEBUG - New I/O worker #1905] (Deferred.java:1330) callback=retry RPC@1048727505 returned Deferred@985517273(state=PENDING, result=null, callback=(continuation of Deferred@2143241706 after retry RPC@1048727505), errback=(continuation of Deferred@2143241706 after retry RPC@1048727505)), so the following Deferred is getting paused: Deferred@2143241706(state=PAUSED, result=Deferred@985517273, callback=org.apache.kudu.client.AsyncKuduClient$6@625add5b -> wakeup thread pool-3-thread-2, errback=passthrough -> wakeup thread pool-3-thread-2) 00:27:40.539 [DEBUG - pool-3-thread-70] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.539 [DEBUG - New I/O worker #1905] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.544 [DEBUG - New I/O worker #1875] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.544 [DEBUG - New I/O worker #1875] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=635), Traces: [0ms] refreshing cache from master, [235ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [235ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [235ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [482ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [635ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.544 [DEBUG - New I/O worker #1875] (Deferred.java:1330) callback=retry RPC@133134819 returned Deferred@64024006(state=PENDING, result=null, callback=(continuation of Deferred@1188197336 after retry RPC@133134819), errback=(continuation of Deferred@1188197336 after retry RPC@133134819)), so the following Deferred is getting paused: Deferred@1188197336(state=PAUSED, result=Deferred@64024006, callback=org.apache.kudu.client.AsyncKuduClient$6@b824082 -> wakeup thread pool-3-thread-72, errback=passthrough -> wakeup thread pool-3-thread-72) 00:27:40.444 [DEBUG - New I/O worker #1804] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.545 [DEBUG - New I/O worker #1804] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=636), Traces: [0ms] refreshing cache from master, [287ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [287ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [287ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [536ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [613ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [616ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:40.545 [DEBUG - New I/O worker #1804] (Deferred.java:1330) callback=retry RPC@672265414 returned Deferred@741242140(state=PENDING, result=null, callback=(continuation of Deferred@1764556541 after retry RPC@672265414), errback=(continuation of Deferred@1764556541 after retry RPC@672265414)), so the following Deferred is getting paused: Deferred@1764556541(state=PAUSED, result=Deferred@741242140, callback=org.apache.kudu.client.AsyncKuduClient$6@7f1fa8ca -> wakeup thread pool-3-thread-59, errback=passthrough -> wakeup thread pool-3-thread-59) 00:27:40.441 [DEBUG - New I/O worker #2022] (Negotiator.java:764) Authenticated connection [id: 0xfd11cff8, /127.0.0.1:53922 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.546 [DEBUG - New I/O worker #1817] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.546 [DEBUG - New I/O worker #1817] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=635), Traces: [0ms] refreshing cache from master, [217ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [217ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [218ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [405ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [620ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [636ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.546 [DEBUG - New I/O worker #1817] (Deferred.java:1330) callback=retry RPC@176142981 returned Deferred@239087378(state=PENDING, result=null, callback=(continuation of Deferred@1262661822 after retry RPC@176142981), errback=(continuation of Deferred@1262661822 after retry RPC@176142981)), so the following Deferred is getting paused: Deferred@1262661822(state=PAUSED, result=Deferred@239087378, callback=org.apache.kudu.client.AsyncKuduClient$6@d1f66e0 -> wakeup thread pool-3-thread-51, errback=passthrough -> wakeup thread pool-3-thread-51) 00:27:40.545 [DEBUG - New I/O worker #1761] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.547 [DEBUG - New I/O worker #1761] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.547 [DEBUG - New I/O worker #1761] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.547 [DEBUG - New I/O worker #1761] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.548 [DEBUG - New I/O worker #1761] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.548 [DEBUG - New I/O worker #1875] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.548 [DEBUG - New I/O worker #1761] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.549 [DEBUG - New I/O worker #1817] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.549 [DEBUG - New I/O worker #1804] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.544 [DEBUG - New I/O worker #1905] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.550 [DEBUG - New I/O worker #1905] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.550 [DEBUG - New I/O worker #1905] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.550 [DEBUG - New I/O worker #1905] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.550 [DEBUG - New I/O worker #1905] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.553 [DEBUG - New I/O worker #1875] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.553 [DEBUG - New I/O worker #1875] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.553 [DEBUG - New I/O worker #1875] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.544 [DEBUG - New I/O worker #1973] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.539 [DEBUG - New I/O worker #1728] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.554 [DEBUG - New I/O worker #1817] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.555 [DEBUG - New I/O worker #1817] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.555 [DEBUG - New I/O worker #1817] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.555 [DEBUG - New I/O worker #1817] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.555 [DEBUG - New I/O worker #1817] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.556 [DEBUG - New I/O worker #1763] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.556 [DEBUG - pool-3-thread-51] (Deferred.java:1330) callback=disconnect callback@63815332 returned Deferred@126482129(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@182878668 after disconnect callback@63815332), errback=passthrough -> (continuation of Deferred@182878668 after disconnect callback@63815332)), so the following Deferred is getting paused: Deferred@182878668(state=PAUSED, result=Deferred@126482129, callback=, errback=) 00:27:40.556 [DEBUG - New I/O worker #2041] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.557 [DEBUG - New I/O worker #2042] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.534 [DEBUG - New I/O worker #1900] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.534 [DEBUG - New I/O worker #1974] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.557 [DEBUG - New I/O worker #1817] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.533 [DEBUG - pool-3-thread-24] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.530 [DEBUG - New I/O worker #1868] (Deferred.java:1330) callback=retry RPC@656895231 returned Deferred@588840131(state=PENDING, result=null, callback=(continuation of Deferred@1713006276 after retry RPC@656895231), errback=(continuation of Deferred@1713006276 after retry RPC@656895231)), so the following Deferred is getting paused: Deferred@1713006276(state=PAUSED, result=Deferred@588840131, callback=org.apache.kudu.client.AsyncKuduClient$6@4b6a28bb -> wakeup thread pool-3-thread-58, errback=passthrough -> wakeup thread pool-3-thread-58) 00:27:40.558 [DEBUG - New I/O worker #1900] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=648), Traces: [0ms] refreshing cache from master, [236ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [236ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [237ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [625ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.528 [DEBUG - New I/O worker #1707] (Deferred.java:1330) callback=retry RPC@168268074 returned Deferred@238552242(state=PENDING, result=null, callback=(continuation of Deferred@1262131985 after retry RPC@168268074), errback=(continuation of Deferred@1262131985 after retry RPC@168268074)), so the following Deferred is getting paused: Deferred@1262131985(state=PAUSED, result=Deferred@238552242, callback=org.apache.kudu.client.AsyncKuduClient$6@4c8343cb -> wakeup thread pool-3-thread-27, errback=passthrough -> wakeup thread pool-3-thread-27) 00:27:40.559 [DEBUG - New I/O worker #1707] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.558 [DEBUG - New I/O worker #1900] (Deferred.java:1330) callback=retry RPC@1782752352 returned Deferred@1853701616(state=PENDING, result=null, callback=(continuation of Deferred@729787995 after retry RPC@1782752352), errback=(continuation of Deferred@729787995 after retry RPC@1782752352)), so the following Deferred is getting paused: Deferred@729787995(state=PAUSED, result=Deferred@1853701616, callback=org.apache.kudu.client.AsyncKuduClient$6@6500dce2 -> wakeup thread pool-3-thread-77, errback=passthrough -> wakeup thread pool-3-thread-77) 00:27:40.558 [DEBUG - New I/O worker #1868] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.555 [DEBUG - New I/O worker #1875] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.560 [DEBUG - New I/O worker #1875] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.553 [DEBUG - New I/O worker #1905] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:40.552917 26336 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.216334 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53954 00:27:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.262058 (+ 45724us) server_negotiation.cc:184] Beginning negotiation 00:27:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.262078 (+ 20us) server_negotiation.cc:373] Waiting for connection header 00:27:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.262117 (+ 39us) server_negotiation.cc:381] Connection header received 00:27:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.262226 (+ 109us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.262251 (+ 25us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.262359 (+ 108us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.268451 (+ 6092us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.467993 (+199542us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.471410 (+ 3417us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.551841 (+ 80431us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53954: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) 00:27:40.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":812,"server-negotiator.queue_time_us":45411,"thread_start_us":6445,"threads_started":1} 00:27:40.548 [DEBUG - pool-3-thread-52] (Deferred.java:1330) callback=disconnect callback@937177182 returned Deferred@1310858232(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1094054207 after disconnect callback@937177182), errback=passthrough -> (continuation of Deferred@1094054207 after disconnect callback@937177182)), so the following Deferred is getting paused: Deferred@1094054207(state=PAUSED, result=Deferred@1310858232, callback=, errback=) 00:27:40.562 [DEBUG - New I/O worker #2021] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.562 [DEBUG - New I/O worker #1875] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.562 [DEBUG - New I/O worker #1868] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.563 [DEBUG - New I/O worker #1868] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.563 [DEBUG - New I/O worker #1868] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.563 [DEBUG - New I/O worker #1868] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.563 [DEBUG - New I/O worker #1868] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.564 [DEBUG - New I/O worker #2022] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:40.564421 26318 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:40.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.146031 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40540 00:27:40.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.180046 (+ 34015us) server_negotiation.cc:184] Beginning negotiation 00:27:40.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.180075 (+ 29us) server_negotiation.cc:373] Waiting for connection header 00:27:40.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.180114 (+ 39us) server_negotiation.cc:381] Connection header received 00:27:40.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.180231 (+ 117us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:40.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.180255 (+ 24us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:40.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.180388 (+ 133us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:40.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.181000 (+ 612us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:40.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.476557 (+295557us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:40.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.484816 (+ 8259us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:40.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.563120 (+ 78304us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40540: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) 00:27:40.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":7041,"server-negotiator.queue_time_us":33733,"thread_start_us":44914,"threads_started":1} 00:27:40.564 [INFO - New I/O worker #2022] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:40.564 [DEBUG - New I/O worker #1868] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.566 [DEBUG - New I/O worker #1900] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.570 [DEBUG - New I/O worker #1804] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.570 [DEBUG - New I/O worker #1804] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.570 [DEBUG - New I/O worker #1804] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.570 [DEBUG - New I/O worker #1804] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.571 [DEBUG - New I/O worker #1804] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.572 [DEBUG - pool-3-thread-59] (Deferred.java:1330) callback=disconnect callback@55756942 returned Deferred@148584467(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1404867390 after disconnect callback@55756942), errback=passthrough -> (continuation of Deferred@1404867390 after disconnect callback@55756942)), so the following Deferred is getting paused: Deferred@1404867390(state=PAUSED, result=Deferred@148584467, callback=, errback=) 00:27:40.572 [DEBUG - New I/O worker #2062] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.572 [DEBUG - New I/O worker #2061] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.573 [DEBUG - New I/O worker #1804] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.574 [DEBUG - New I/O worker #1900] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.574 [DEBUG - New I/O worker #1900] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.574 [DEBUG - New I/O worker #1900] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.574 [DEBUG - New I/O worker #1900] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.574 [DEBUG - New I/O worker #1900] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.575 [DEBUG - New I/O worker #1854] (Negotiator.java:764) Authenticated connection [id: 0x13ea8ab3, /127.0.0.1:38494 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:40.577000 26321 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.147816 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53916 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.184113 (+ 36297us) server_negotiation.cc:184] Beginning negotiation 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.184149 (+ 36us) server_negotiation.cc:373] Waiting for connection header 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.184201 (+ 52us) server_negotiation.cc:381] Connection header received 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.184316 (+ 115us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.184337 (+ 21us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.184440 (+ 103us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.184838 (+ 398us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.476698 (+291860us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.477416 (+ 718us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.575957 (+ 98541us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53916: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) 00:27:40.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":35979,"thread_start_us":35830,"threads_started":1} 00:27:40.578 [DEBUG - New I/O worker #1900] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.578 [DEBUG - New I/O worker #1854] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.579 [DEBUG - New I/O worker #1854] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=627), Traces: [0ms] refreshing cache from master, [156ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [156ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [157ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [393ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [627ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.579 [DEBUG - New I/O worker #1974] (Negotiator.java:764) Authenticated connection [id: 0x76b7b74d, /127.0.0.1:40596 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.579 [DEBUG - New I/O worker #1854] (Deferred.java:1330) callback=retry RPC@223925951 returned Deferred@157762508(state=PENDING, result=null, callback=(continuation of Deferred@1281744004 after retry RPC@223925951), errback=(continuation of Deferred@1281744004 after retry RPC@223925951)), so the following Deferred is getting paused: Deferred@1281744004(state=PAUSED, result=Deferred@157762508, callback=org.apache.kudu.client.AsyncKuduClient$6@3645d77c -> wakeup thread pool-3-thread-29, errback=passthrough -> wakeup thread pool-3-thread-29) 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:40.579942 26353 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.191948 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40552 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.210841 (+ 18893us) server_negotiation.cc:184] Beginning negotiation 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.210870 (+ 29us) server_negotiation.cc:373] Waiting for connection header 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.210907 (+ 37us) server_negotiation.cc:381] Connection header received 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.211028 (+ 121us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.211054 (+ 26us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.211178 (+ 124us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.214579 (+ 3401us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.308095 (+ 93516us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.310015 (+ 1920us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.468745 (+158730us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.469989 (+ 1244us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.470207 (+ 218us) server_negotiation.cc:589] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.566802 (+ 96595us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40552: BlockingRecv error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) 00:27:40.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":18608,"thread_start_us":23328,"threads_started":1} 00:27:40.581 [DEBUG - New I/O worker #1821] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.582 [DEBUG - pool-3-thread-18] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.583 [DEBUG - New I/O worker #1854] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.585 [DEBUG - New I/O worker #1728] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.585 [DEBUG - New I/O worker #1728] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.585 [DEBUG - New I/O worker #1728] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.585 [DEBUG - New I/O worker #1728] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.586 [DEBUG - New I/O worker #1728] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.586 [DEBUG - New I/O worker #1854] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.586 [DEBUG - New I/O worker #1854] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.587 [DEBUG - New I/O worker #2075] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.586 [DEBUG - pool-3-thread-71] (Deferred.java:1330) callback=disconnect callback@1683513556 returned Deferred@384159507(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1055599895 after disconnect callback@1683513556), errback=passthrough -> (continuation of Deferred@1055599895 after disconnect callback@1683513556)), so the following Deferred is getting paused: Deferred@1055599895(state=PAUSED, result=Deferred@384159507, callback=, errback=) 00:27:40.587 [DEBUG - New I/O worker #1854] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.587 [DEBUG - New I/O worker #2076] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.587 [DEBUG - New I/O worker #1854] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.588 [DEBUG - New I/O worker #1854] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.588 [DEBUG - New I/O worker #1728] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.589 [DEBUG - pool-3-thread-29] (Deferred.java:1330) callback=disconnect callback@199071641 returned Deferred@2088528580(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@634329039 after disconnect callback@199071641), errback=passthrough -> (continuation of Deferred@634329039 after disconnect callback@199071641)), so the following Deferred is getting paused: Deferred@634329039(state=PAUSED, result=Deferred@2088528580, callback=, errback=) 00:27:40.589 [DEBUG - New I/O worker #1856] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.589 [DEBUG - New I/O worker #1854] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:40.591763 22628 connection.cc:554] Connection torn down before Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:53922 (request call id 0) could send its response 00:27:40.597 [DEBUG - New I/O worker #1996] (Negotiator.java:764) Authenticated connection [id: 0x4ec4e0a2, /127.0.0.1:40608 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.601 [DEBUG - New I/O worker #1882] (Negotiator.java:764) Authenticated connection [id: 0x7d8b3894, /127.0.0.1:38632 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.602 [DEBUG - New I/O worker #1884] (Negotiator.java:764) Authenticated connection [id: 0xb54e6b7f, /127.0.0.1:53978 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.603 [DEBUG - New I/O worker #1896] (Negotiator.java:764) Authenticated connection [id: 0x1a9e4136, /127.0.0.1:38620 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.603 [DEBUG - New I/O worker #1707] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.603 [DEBUG - New I/O worker #1707] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.603 [DEBUG - New I/O worker #1707] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.604 [DEBUG - New I/O worker #1707] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.604 [DEBUG - New I/O worker #1707] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.605 [DEBUG - pool-3-thread-27] (Deferred.java:1330) callback=disconnect callback@1699602085 returned Deferred@704273799(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@274549239 after disconnect callback@1699602085), errback=passthrough -> (continuation of Deferred@274549239 after disconnect callback@1699602085)), so the following Deferred is getting paused: Deferred@274549239(state=PAUSED, result=Deferred@704273799, callback=, errback=) 00:27:40.605 [DEBUG - New I/O worker #2083] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.606 [DEBUG - New I/O worker #1896] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.606 [DEBUG - New I/O worker #1709] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.606 [DEBUG - New I/O worker #1707] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.608 [DEBUG - New I/O worker #1923] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.609 [DEBUG - New I/O worker #1973] (Negotiator.java:764) Authenticated connection [id: 0x9cb173b6, /127.0.0.1:53966 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.606 [DEBUG - New I/O worker #1896] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=535), Traces: [0ms] refreshing cache from master, [255ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [378ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [379ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [511ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [535ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.609 [DEBUG - New I/O worker #1896] (Deferred.java:1330) callback=retry RPC@1266652658 returned Deferred@1329600581(state=PENDING, result=null, callback=(continuation of Deferred@172138441 after retry RPC@1266652658), errback=(continuation of Deferred@172138441 after retry RPC@1266652658)), so the following Deferred is getting paused: Deferred@172138441(state=PAUSED, result=Deferred@1329600581, callback=org.apache.kudu.client.AsyncKuduClient$6@c65646f -> wakeup thread pool-3-thread-35, errback=passthrough -> wakeup thread pool-3-thread-35) 00:27:40.616 [DEBUG - New I/O worker #1882] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.616 [DEBUG - New I/O worker #1882] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=527), Traces: [0ms] refreshing cache from master, [367ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [367ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [367ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [514ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [516ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [526ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.616 [DEBUG - New I/O worker #1930] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.617 [DEBUG - New I/O worker #1882] (Deferred.java:1330) callback=retry RPC@777556050 returned Deferred@711393601(state=PENDING, result=null, callback=(continuation of Deferred@1868929641 after retry RPC@777556050), errback=(continuation of Deferred@1868929641 after retry RPC@777556050)), so the following Deferred is getting paused: Deferred@1868929641(state=PAUSED, result=Deferred@711393601, callback=org.apache.kudu.client.AsyncKuduClient$6@3470ab56 -> wakeup thread pool-3-thread-55, errback=passthrough -> wakeup thread pool-3-thread-55) 00:27:40.623 [DEBUG - New I/O worker #1882] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.623 [DEBUG - New I/O worker #1998] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.628 [DEBUG - New I/O worker #1930] (Negotiator.java:764) Authenticated connection [id: 0x1d54da2b, /127.0.0.1:53960 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.629 [DEBUG - New I/O worker #1882] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.629 [DEBUG - New I/O worker #1882] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.629 [DEBUG - New I/O worker #1882] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.630 [DEBUG - New I/O worker #1882] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.630 [DEBUG - New I/O worker #1882] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.631 [DEBUG - New I/O worker #1884] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.632 [DEBUG - New I/O worker #1882] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.639 [DEBUG - New I/O worker #1901] (Negotiator.java:764) Authenticated connection [id: 0x993236f4, /127.0.0.1:38582 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.639 [DEBUG - New I/O worker #2028] (Negotiator.java:764) Authenticated connection [id: 0x367163f6, /127.0.0.1:40572 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.642 [DEBUG - New I/O worker #1901] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.642 [DEBUG - New I/O worker #1901] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=733), Traces: [0ms] refreshing cache from master, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [263ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [421ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [733ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.643 [DEBUG - New I/O worker #1885] (Deferred.java:1330) callback=retry RPC@758422462 returned Deferred@688589504(state=PENDING, result=null, callback=(continuation of Deferred@1812566405 after retry RPC@758422462), errback=(continuation of Deferred@1812566405 after retry RPC@758422462)), so the following Deferred is getting paused: Deferred@1812566405(state=PAUSED, result=Deferred@688589504, callback=org.apache.kudu.client.AsyncKuduClient$6@544f9355 -> wakeup thread Time-limited test, errback=passthrough -> wakeup thread Time-limited test) 00:27:40.652 [DEBUG - New I/O worker #1892] (Negotiator.java:764) Authenticated connection [id: 0x2a309fc6, /127.0.0.1:38602 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.651 [DEBUG - New I/O worker #1762] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.656 [DEBUG - New I/O worker #2108] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.657 [DEBUG - New I/O worker #1958] (Negotiator.java:764) Authenticated connection [id: 0xf3880db2, /127.0.0.1:40546 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.658 [DEBUG - New I/O worker #1901] (Deferred.java:1330) callback=retry RPC@1008337911 returned Deferred@941981868(state=PENDING, result=null, callback=(continuation of Deferred@2099710412 after retry RPC@1008337911), errback=(continuation of Deferred@2099710412 after retry RPC@1008337911)), so the following Deferred is getting paused: Deferred@2099710412(state=PAUSED, result=Deferred@941981868, callback=org.apache.kudu.client.AsyncKuduClient$6@20f98ebf -> wakeup thread pool-3-thread-39, errback=passthrough -> wakeup thread pool-3-thread-39) 00:27:40.660 [DEBUG - New I/O worker #1901] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:40.649385 22690 connection.cc:320] Error closing socket: Network error: TlsSocket::Close: Connection reset by peer 00:27:40.648 [DEBUG - New I/O worker #1857] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.647 [DEBUG - New I/O worker #1898] (Negotiator.java:764) Authenticated connection [id: 0xfb4c4d4e, /127.0.0.1:38558 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.647 [DEBUG - New I/O worker #2163] (Negotiator.java:764) Authenticated connection [id: 0xb5a22810, /127.0.0.1:40578 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.662 [DEBUG - New I/O worker #1923] (Negotiator.java:764) Authenticated connection [id: 0x882a8914, /127.0.0.1:38614 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.645 [DEBUG - New I/O worker #1996] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.643 [DEBUG - pool-3-thread-55] (Deferred.java:1330) callback=disconnect callback@388916452 returned Deferred@168574271(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1090879854 after disconnect callback@388916452), errback=passthrough -> (continuation of Deferred@1090879854 after disconnect callback@388916452)), so the following Deferred is getting paused: Deferred@1090879854(state=PAUSED, result=Deferred@168574271, callback=, errback=) 00:27:40.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:40.661267 26096 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.124744 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53866 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.125127 (+ 383us) server_negotiation.cc:184] Beginning negotiation 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.125149 (+ 22us) server_negotiation.cc:373] Waiting for connection header 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.125204 (+ 55us) server_negotiation.cc:381] Connection header received 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.125308 (+ 104us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.125333 (+ 25us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.125451 (+ 118us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.125615 (+ 164us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.294953 (+169338us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.296143 (+ 1190us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.451807 (+155664us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.453401 (+ 1594us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:40.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.453662 (+ 261us) server_negotiation.cc:589] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 00:27:40.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.468362 (+ 14700us) server_negotiation.cc:337] Received SASL_INITIATE NegotiatePB request 00:27:40.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.468389 (+ 27us) server_negotiation.cc:746] Received SASL_INITIATE request from client 00:27:40.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.468411 (+ 22us) server_negotiation.cc:757] Client requested to use mechanism: PLAIN 00:27:40.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.468468 (+ 57us) server_negotiation.cc:786] Calling sasl_server_start() 00:27:40.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.468533 (+ 65us) server_negotiation.cc:945] Received PLAIN auth, user=slave 00:27:40.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.468574 (+ 41us) server_negotiation.cc:349] Sending SASL_SUCCESS NegotiatePB response 00:27:40.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.468771 (+ 197us) server_negotiation.cc:890] Waiting for connection context 00:27:40.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.651999 (+183228us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53866: BlockingRecv error: failed to read from TLS socket (remote: 0.0.0.0:0): Cannot send after transport endpoint shutdown (error 108) 00:27:40.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":94} 00:27:40.664 [DEBUG - New I/O worker #1901] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.664 [DEBUG - New I/O worker #1901] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.653 [DEBUG - New I/O worker #1885] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.666 [DEBUG - New I/O worker #1923] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.667 [DEBUG - New I/O worker #1923] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=588), Traces: [0ms] refreshing cache from master, [246ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [369ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [369ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [560ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [586ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.667 [DEBUG - New I/O worker #1923] (Deferred.java:1330) callback=retry RPC@1862410196 returned Deferred@1799198444(state=PENDING, result=null, callback=(continuation of Deferred@775886319 after retry RPC@1862410196), errback=(continuation of Deferred@775886319 after retry RPC@1862410196)), so the following Deferred is getting paused: Deferred@775886319(state=PAUSED, result=Deferred@1799198444, callback=org.apache.kudu.client.AsyncKuduClient$6@60d824a9 -> wakeup thread pool-3-thread-10, errback=passthrough -> wakeup thread pool-3-thread-10) 00:27:40.669 [DEBUG - New I/O worker #1923] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.671 [DEBUG - New I/O worker #1898] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.671 [DEBUG - New I/O worker #1898] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=762), Traces: [0ms] refreshing cache from master, [232ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [233ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [233ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [660ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [762ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.672 [DEBUG - New I/O worker #1898] (Deferred.java:1330) callback=retry RPC@755655124 returned Deferred@691396781(state=PENDING, result=null, callback=(continuation of Deferred@1815571439 after retry RPC@755655124), errback=(continuation of Deferred@1815571439 after retry RPC@755655124)), so the following Deferred is getting paused: Deferred@1815571439(state=PAUSED, result=Deferred@691396781, callback=org.apache.kudu.client.AsyncKuduClient$6@236b96d7 -> wakeup thread pool-3-thread-57, errback=passthrough -> wakeup thread pool-3-thread-57) 00:27:40.671 [DEBUG - New I/O worker #1885] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.672 [DEBUG - New I/O worker #1885] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.672 [DEBUG - New I/O worker #1885] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.672 [DEBUG - New I/O worker #1] (Connection.java:429) [peer master-127.22.19.252:32833(127.22.19.252:32833)] encountered a read timeout; closing the channel 00:27:40.672 [DEBUG - New I/O worker #1885] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.677 [DEBUG - New I/O worker #1986] (Negotiator.java:764) Authenticated connection [id: 0xd67cc09e, /127.0.0.1:53898 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.673 [DEBUG - New I/O worker #1] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: [peer master-127.22.19.252:32833(127.22.19.252:32833)] encountered a read timeout; closing the channel 00:27:40.680 [DEBUG - New I/O worker #1896] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.679 [DEBUG - New I/O worker #1901] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.686 [DEBUG - New I/O worker #1920] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.686 [DEBUG - New I/O worker #2071] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.686 [DEBUG - New I/O worker #2109] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.683 [DEBUG - New I/O worker #1892] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.682 [DEBUG - New I/O worker #1898] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.680 [DEBUG - New I/O worker #1965] (Negotiator.java:764) Authenticated connection [id: 0xf72e4a68, /127.0.0.1:53904 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.711 [DEBUG - New I/O worker #2090] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.718 [DEBUG - New I/O worker #1885] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299992, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299990, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299975, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.732 [DEBUG - New I/O worker #1986] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.707 [DEBUG - New I/O worker #1923] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.707 [DEBUG - New I/O worker #1896] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.702 [DEBUG - New I/O worker #1728] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.699 [DEBUG - New I/O worker #1901] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.764 [DEBUG - New I/O worker #1901] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.693 [DEBUG - New I/O worker #1850] (Negotiator.java:764) Authenticated connection [id: 0x81c1d34f, /127.0.0.1:38530 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.692 [DEBUG - New I/O worker #1806] (Negotiator.java:764) Authenticated connection [id: 0x7681cd91, /127.0.0.1:38498 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.689 [DEBUG - New I/O worker #1857] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.765 [INFO - New I/O worker #1857] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:40.765 [DEBUG - New I/O worker #1901] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.763 [DEBUG - New I/O worker #1896] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.763 [DEBUG - New I/O worker #1887] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.768 [DEBUG - New I/O worker #1896] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.763 [DEBUG - New I/O worker #1923] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.763 [DEBUG - New I/O worker #1994] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.762 [DEBUG - New I/O worker #1739] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.762 [DEBUG - New I/O worker #2067] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.762 [DEBUG - New I/O worker #2138] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.762 [DEBUG - New I/O worker #1985] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.811 [DEBUG - New I/O worker #2086] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.762 [DEBUG - New I/O worker #2074] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.762 [DEBUG - New I/O worker #2136] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.762 [DEBUG - New I/O worker #1995] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.762 [DEBUG - New I/O worker #2091] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.762 [DEBUG - New I/O worker #1978] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.762 [DEBUG - New I/O worker #1797] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.762 [DEBUG - New I/O worker #2157] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.762 [DEBUG - New I/O worker #2072] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.762 [DEBUG - New I/O worker #1890] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.762 [DEBUG - New I/O worker #2073] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.762 [DEBUG - New I/O worker #1740] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.762 [DEBUG - New I/O worker #1744] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.762 [DEBUG - New I/O worker #1916] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.762 [DEBUG - New I/O worker #2070] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.762 [DEBUG - New I/O worker #1894] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.762 [DEBUG - New I/O worker #2110] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.762 [DEBUG - New I/O worker #1742] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.762 [DEBUG - New I/O worker #1817] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.759 [DEBUG - New I/O worker #1958] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.758 [DEBUG - New I/O worker #1957] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:40.853 [INFO - New I/O worker #1957] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:40.758 [DEBUG - Time-limited test] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.758 [DEBUG - New I/O worker #2023] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:40.853 [INFO - New I/O worker #2023] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:40.758 [DEBUG - New I/O worker #2096] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.758 [DEBUG - pool-3-thread-77] (Deferred.java:1330) callback=disconnect callback@1829379309 returned Deferred@1991938449(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@279970477 after disconnect callback@1829379309), errback=passthrough -> (continuation of Deferred@279970477 after disconnect callback@1829379309)), so the following Deferred is getting paused: Deferred@279970477(state=PAUSED, result=Deferred@1991938449, callback=, errback=) 00:27:40.758 [DEBUG - New I/O worker #1949] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:40.854 [INFO - New I/O worker #1949] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:40.758 [DEBUG - pool-3-thread-2] (Deferred.java:1330) callback=disconnect callback@2049514704 returned Deferred@1461098389(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@489862608 after disconnect callback@2049514704), errback=passthrough -> (continuation of Deferred@489862608 after disconnect callback@2049514704)), so the following Deferred is getting paused: Deferred@489862608(state=PAUSED, result=Deferred@1461098389, callback=, errback=) 00:27:40.758 [DEBUG - New I/O worker #2148] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:40.758 [DEBUG - pool-3-thread-58] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.758 [DEBUG - pool-3-thread-72] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.734 [DEBUG - New I/O worker #2004] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.733 [DEBUG - New I/O worker #1804] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:40.730412 26312 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.138843 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40528 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.142812 (+ 3969us) server_negotiation.cc:184] Beginning negotiation 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.142839 (+ 27us) server_negotiation.cc:373] Waiting for connection header 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.142873 (+ 34us) server_negotiation.cc:381] Connection header received 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.142996 (+ 123us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.143017 (+ 21us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.143135 (+ 118us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.211618 (+ 68483us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.323145 (+111527us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.486745 (+163600us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.729083 (+242338us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40528: BlockingRecv error: recv got EOF from 127.0.0.1:40528 (error 108) 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":9854,"server-negotiator.queue_time_us":3722,"thread_start_us":3542,"threads_started":1} 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:40.767314 26320 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.173361 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40558 00:27:40.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.208509 (+ 35148us) server_negotiation.cc:184] Beginning negotiation 00:27:40.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.208538 (+ 29us) server_negotiation.cc:373] Waiting for connection header 00:27:40.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.208574 (+ 36us) server_negotiation.cc:381] Connection header received 00:27:40.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.208663 (+ 89us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:40.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.208683 (+ 20us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:40.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.208882 (+ 199us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:40.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.209040 (+ 158us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:40.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.477930 (+268890us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:40.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.487209 (+ 9279us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:40.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:40.766239 (+279030us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40558: BlockingRecv error: recv got EOF from 127.0.0.1:40558 (error 108) 00:27:40.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":7515,"server-negotiator.queue_time_us":34890,"thread_start_us":31039,"threads_started":1} 00:27:40.724 [DEBUG - New I/O worker #1898] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.724 [DEBUG - New I/O worker #1892] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=699), Traces: [0ms] refreshing cache from master, [200ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [200ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [582ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [657ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [670ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.723 [DEBUG - New I/O worker #2058] (Negotiator.java:764) Authenticated connection [id: 0xb3cca288, /127.0.0.1:53844 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.719 [DEBUG - New I/O worker #1931] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.868 [DEBUG - New I/O worker #1898] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:40.860 [INFO - New I/O worker #2148] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:40.811 [DEBUG - New I/O worker #2085] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.811 [DEBUG - New I/O worker #1706] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.810 [DEBUG - pool-3-thread-15] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.810 [DEBUG - New I/O worker #1923] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:40.806 [DEBUG - New I/O worker #1896] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.803 [DEBUG - pool-3-thread-14] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:40.799 [DEBUG - New I/O worker #1762] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:40.894 [INFO - New I/O worker #1762] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:40.895 [DEBUG - New I/O worker #1887] (Negotiator.java:764) Authenticated connection [id: 0x4b6aa023, /127.0.0.1:38676 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.798 [DEBUG - New I/O worker #2071] (Negotiator.java:764) Authenticated connection [id: 0x7047afb9, /127.0.0.1:40622 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.795 [DEBUG - New I/O worker #1707] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.795 [DEBUG - New I/O worker #2108] (Negotiator.java:764) Authenticated connection [id: 0x25d45762, /127.0.0.1:40590 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.791 [DEBUG - New I/O worker #2109] (Negotiator.java:764) Authenticated connection [id: 0xb6a7aa38, /127.0.0.1:53986 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.787 [DEBUG - New I/O worker #1996] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:40.784 [DEBUG - New I/O worker #1920] (Negotiator.java:764) Authenticated connection [id: 0x6fd5681c, /127.0.0.1:38640 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.780 [DEBUG - New I/O worker #1695] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:40.780 [DEBUG - New I/O worker #1696] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.779 [DEBUG - New I/O worker #1694] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.776 [INFO - New I/O worker #1987] (Connection.java:435) [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:40.915 [DEBUG - New I/O worker #1987] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:40.776 [DEBUG - New I/O worker #1932] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.776 [DEBUG - New I/O worker #1806] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.923 [DEBUG - New I/O worker #1806] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=972), Traces: [0ms] refreshing cache from master, [158ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [158ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [158ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [625ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [818ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [926ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:40.775 [DEBUG - New I/O worker #1850] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.775 [DEBUG - New I/O worker #1979] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:40.773 [DEBUG - New I/O worker #1736] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:40.771 [INFO - New I/O worker #1951] (Connection.java:435) [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:40.771 [DEBUG - New I/O worker #1885] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:40.771 [DEBUG - pool-3-thread-39] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:40.765 [DEBUG - New I/O worker #1903] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:40.943 [DEBUG - New I/O worker #1850] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1016), Traces: [0ms] refreshing cache from master, [213ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [213ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [297ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [396ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [733ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [858ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.952 [DEBUG - New I/O worker #1850] (Deferred.java:1330) callback=retry RPC@785271471 returned Deferred@720489390(state=PENDING, result=null, callback=(continuation of Deferred@1878212756 after retry RPC@785271471), errback=(continuation of Deferred@1878212756 after retry RPC@785271471)), so the following Deferred is getting paused: Deferred@1878212756(state=PAUSED, result=Deferred@720489390, callback=org.apache.kudu.client.AsyncKuduClient$6@21112e4 -> wakeup thread pool-3-thread-23, errback=passthrough -> wakeup thread pool-3-thread-23) 00:27:40.934 [DEBUG - New I/O worker #1806] (Deferred.java:1330) callback=retry RPC@359399247 returned Deferred@290744634(state=PENDING, result=null, callback=(continuation of Deferred@1414985076 after retry RPC@359399247), errback=(continuation of Deferred@1414985076 after retry RPC@359399247)), so the following Deferred is getting paused: Deferred@1414985076(state=PAUSED, result=Deferred@290744634, callback=org.apache.kudu.client.AsyncKuduClient$6@2f9303e6 -> wakeup thread pool-3-thread-76, errback=passthrough -> wakeup thread pool-3-thread-76) 00:27:40.953 [DEBUG - New I/O worker #1806] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.934 [DEBUG - New I/O worker #1951] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:40.958 [INFO - New I/O worker #1951] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:40.933 [DEBUG - New I/O worker #2073] (Negotiator.java:764) Authenticated connection [id: 0x8d89cfad, /127.0.0.1:53972 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.930 [DEBUG - New I/O worker #2090] (Negotiator.java:764) Authenticated connection [id: 0x35632e23, /127.0.0.1:54016 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.927 [DEBUG - New I/O worker #1923] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:40.926 [DEBUG - New I/O worker #1920] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.925 [DEBUG - New I/O worker #1739] (Negotiator.java:764) Authenticated connection [id: 0x9a06f89b, /127.0.0.1:38646 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.922 [INFO - New I/O worker #1987] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:40.921 [DEBUG - New I/O worker #2136] (Negotiator.java:764) Authenticated connection [id: 0xeedba4ea, /127.0.0.1:40646 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.921 [DEBUG - New I/O worker #2086] (Negotiator.java:764) Authenticated connection [id: 0xb8300507, /127.0.0.1:40676 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.921 [DEBUG - New I/O worker #1994] (Negotiator.java:764) Authenticated connection [id: 0x9180643a, /127.0.0.1:54022 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.921 [DEBUG - New I/O worker #1744] (Negotiator.java:764) Authenticated connection [id: 0xd4f22a6c, /127.0.0.1:38660 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.897 [DEBUG - New I/O worker #2157] (Negotiator.java:764) Authenticated connection [id: 0x56f66625, /127.0.0.1:40658 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.896 [DEBUG - New I/O worker #1797] (Negotiator.java:764) Authenticated connection [id: 0xb7b29ce7, /127.0.0.1:38688 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.896 [DEBUG - New I/O worker #2067] (Negotiator.java:764) Authenticated connection [id: 0x17296e4f, /127.0.0.1:54034 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.894 [DEBUG - New I/O worker #1896] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299996, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299994, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299994, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299994, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:40.885 [DEBUG - New I/O worker #1892] (Deferred.java:1330) callback=retry RPC@1917821517 returned Deferred@1987061980(state=PENDING, result=null, callback=(continuation of Deferred@863152246 after retry RPC@1917821517), errback=(continuation of Deferred@863152246 after retry RPC@1917821517)), so the following Deferred is getting paused: Deferred@863152246(state=PAUSED, result=Deferred@1987061980, callback=org.apache.kudu.client.AsyncKuduClient$6@d591897 -> wakeup thread pool-3-thread-67, errback=passthrough -> wakeup thread pool-3-thread-67) 00:27:40.884 [DEBUG - New I/O worker #1898] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.012 [DEBUG - New I/O worker #1744] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.012 [DEBUG - New I/O worker #1739] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.012 [DEBUG - New I/O worker #1979] (Negotiator.java:764) Authenticated connection [id: 0x2abb2931, /127.0.0.1:40628 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.016 [DEBUG - New I/O worker #1739] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=432), Traces: [0ms] refreshing cache from master, [97ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [98ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [98ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [343ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [418ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:41.016 [DEBUG - New I/O worker #1739] (Deferred.java:1330) callback=retry RPC@979589057 returned Deferred@1046271061(state=PENDING, result=null, callback=(continuation of Deferred@2069782010 after retry RPC@979589057), errback=(continuation of Deferred@2069782010 after retry RPC@979589057)), so the following Deferred is getting paused: Deferred@2069782010(state=PAUSED, result=Deferred@1046271061, callback=org.apache.kudu.client.AsyncKuduClient$6@6a929823 -> wakeup thread pool-3-thread-18, errback=passthrough -> wakeup thread pool-3-thread-18) 00:27:41.019 [DEBUG - New I/O worker #1932] (Negotiator.java:764) Authenticated connection [id: 0x1d80d380, /127.0.0.1:40638 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.011 [DEBUG - New I/O worker #1696] (Negotiator.java:764) Authenticated connection [id: 0x2963a05a, /127.0.0.1:40670 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.011 [DEBUG - New I/O worker #1896] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.011 [DEBUG - New I/O worker #1973] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.011 [DEBUG - New I/O worker #1923] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299989, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299981, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299979, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299973, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.010 [DEBUG - New I/O worker #1974] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.010 [DEBUG - New I/O worker #1736] (Negotiator.java:764) Authenticated connection [id: 0xc534b892, /127.0.0.1:38626 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.010 [DEBUG - pool-3-thread-35] (Deferred.java:1330) callback=disconnect callback@1490955998 returned Deferred@172409591(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1491658716 after disconnect callback@1490955998), errback=passthrough -> (continuation of Deferred@1491658716 after disconnect callback@1490955998)), so the following Deferred is getting paused: Deferred@1491658716(state=PAUSED, result=Deferred@172409591, callback=, errback=) 00:27:41.007 [DEBUG - New I/O worker #1694] (Negotiator.java:764) Authenticated connection [id: 0x241cc7b1, /127.0.0.1:38694 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.024 [DEBUG - New I/O worker #1695] (Negotiator.java:764) Authenticated connection [id: 0x17428723, /127.0.0.1:54040 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.007 [DEBUG - New I/O worker #1931] (Negotiator.java:764) Authenticated connection [id: 0x982f8d6f, /127.0.0.1:54006 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.024 [DEBUG - New I/O worker #1736] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.024 [DEBUG - New I/O worker #1736] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=955), Traces: [0ms] refreshing cache from master, [385ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [386ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [386ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [912ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [955ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.994 [DEBUG - New I/O worker #1706] (Negotiator.java:764) Authenticated connection [id: 0x2c7912a5, /127.0.0.1:38700 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.983 [DEBUG - New I/O worker #2085] (Negotiator.java:764) Authenticated connection [id: 0xf8a6e463, /127.0.0.1:54046 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.027 [DEBUG - New I/O worker #2096] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.027 [DEBUG - New I/O worker #1706] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.028 [DEBUG - New I/O worker #1706] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=469), Traces: [0ms] refreshing cache from master, [244ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [245ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [245ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [438ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [469ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.983 [DEBUG - New I/O worker #1742] (Negotiator.java:764) Authenticated connection [id: 0x6bd20dd3, /127.0.0.1:53992 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.029 [DEBUG - New I/O worker #1916] (Negotiator.java:764) Authenticated connection [id: 0x5ebc3e39, /127.0.0.1:38658 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.030 [DEBUG - New I/O worker #1706] (Deferred.java:1330) callback=retry RPC@1382967339 returned Deferred@1448212158(state=PENDING, result=null, callback=(continuation of Deferred@324231696 after retry RPC@1382967339), errback=(continuation of Deferred@324231696 after retry RPC@1382967339)), so the following Deferred is getting paused: Deferred@324231696(state=PAUSED, result=Deferred@1448212158, callback=org.apache.kudu.client.AsyncKuduClient$6@304da59 -> wakeup thread pool-3-thread-24, errback=passthrough -> wakeup thread pool-3-thread-24) 00:27:41.031 [DEBUG - New I/O worker #1890] (Negotiator.java:764) Authenticated connection [id: 0x8bdcbdf6, /127.0.0.1:38682 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.031 [DEBUG - New I/O worker #1706] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.982 [DEBUG - New I/O worker #1740] (Negotiator.java:764) Authenticated connection [id: 0xd8b002be, /127.0.0.1:38670 => /127.22.19.252:32833] using SASL/PLAIN 00:27:40.982 [DEBUG - New I/O worker #1985] (Negotiator.java:764) Authenticated connection [id: 0x7f1dc515, /127.0.0.1:54028 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.981 [DEBUG - New I/O worker #1806] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:40.981 [DEBUG - New I/O worker #1920] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=758), Traces: [0ms] refreshing cache from master, [456ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [457ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [457ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [692ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [696ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:40.979 [DEBUG - New I/O worker #1978] (Negotiator.java:764) Authenticated connection [id: 0x96fe6833, /127.0.0.1:53998 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.977 [DEBUG - New I/O worker #2070] (Negotiator.java:764) Authenticated connection [id: 0x541dac15, /127.0.0.1:54008 => /127.22.19.254:44849] using SASL/PLAIN 00:27:40.967 [DEBUG - New I/O worker #1850] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:40.962 [DEBUG - New I/O worker #1995] (Negotiator.java:764) Authenticated connection [id: 0xe4daa2a0, /127.0.0.1:40652 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.960 [DEBUG - New I/O worker #2091] (Negotiator.java:764) Authenticated connection [id: 0x2643c188, /127.0.0.1:40664 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.947 [DEBUG - New I/O worker #1887] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:40.946 [DEBUG - New I/O worker #2074] (Negotiator.java:764) Authenticated connection [id: 0xac853cac, /127.0.0.1:40602 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.946 [DEBUG - New I/O worker #2138] (Negotiator.java:764) Authenticated connection [id: 0x86dde9b0, /127.0.0.1:40640 => /127.22.19.253:37879] using SASL/PLAIN 00:27:40.945 [DEBUG - New I/O worker #2110] (Negotiator.java:764) Authenticated connection [id: 0xd2e171a1, /127.0.0.1:40616 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.058 [DEBUG - New I/O worker #1927] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.058 [DEBUG - New I/O worker #2089] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.051 [DEBUG - New I/O worker #1894] (Negotiator.java:764) Authenticated connection [id: 0x4bccbd11, /127.0.0.1:38652 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.042 [DEBUG - New I/O worker #1916] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.066 [DEBUG - New I/O worker #1916] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=844), Traces: [0ms] refreshing cache from master, [469ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [469ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [469ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [799ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [810ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [812ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:41.040 [DEBUG - New I/O worker #1920] (Deferred.java:1330) callback=retry RPC@593728787 returned Deferred@660342183(state=PENDING, result=null, callback=(continuation of Deferred@1650368296 after retry RPC@593728787), errback=(continuation of Deferred@1650368296 after retry RPC@593728787)), so the following Deferred is getting paused: Deferred@1650368296(state=PAUSED, result=Deferred@660342183, callback=org.apache.kudu.client.AsyncKuduClient$6@6c828762 -> wakeup thread pool-3-thread-80, errback=passthrough -> wakeup thread pool-3-thread-80) 00:27:41.067 [DEBUG - New I/O worker #1920] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.069 [DEBUG - New I/O worker #1916] (Deferred.java:1330) callback=retry RPC@214373137 returned Deferred@150505041(state=PENDING, result=null, callback=(continuation of Deferred@1308236074 after retry RPC@214373137), errback=(continuation of Deferred@1308236074 after retry RPC@214373137)), so the following Deferred is getting paused: Deferred@1308236074(state=PAUSED, result=Deferred@150505041, callback=org.apache.kudu.client.AsyncKuduClient$6@48968c51 -> wakeup thread pool-3-thread-64, errback=passthrough -> wakeup thread pool-3-thread-64) 00:27:41.039 [DEBUG - New I/O worker #1806] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.071 [DEBUG - New I/O worker #1916] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.038 [DEBUG - New I/O worker #1706] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.075 [DEBUG - New I/O worker #1706] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.075 [DEBUG - New I/O worker #1706] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.076 [DEBUG - New I/O worker #1706] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.038 [DEBUG - New I/O worker #1740] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.037 [DEBUG - New I/O worker #1890] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.079 [DEBUG - New I/O worker #1890] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=856), Traces: [0ms] refreshing cache from master, [474ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [475ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [475ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [785ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [813ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [822ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:41.079 [DEBUG - New I/O worker #1706] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.034 [DEBUG - New I/O worker #4] (Connection.java:429) [peer c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)] encountered a read timeout; closing the channel 00:27:41.080 [DEBUG - New I/O worker #4] (Connection.java:688) [peer c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)] cleaning up while in state READY due to: [peer c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)] encountered a read timeout; closing the channel 00:27:41.033 [DEBUG - New I/O worker #1958] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.028 [DEBUG - New I/O worker #1896] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.025 [DEBUG - New I/O worker #1736] (Deferred.java:1330) callback=retry RPC@102591618 returned Deferred@35846297(state=PENDING, result=null, callback=(continuation of Deferred@1193305785 after retry RPC@102591618), errback=(continuation of Deferred@1193305785 after retry RPC@102591618)), so the following Deferred is getting paused: Deferred@1193305785(state=PAUSED, result=Deferred@35846297, callback=org.apache.kudu.client.AsyncKuduClient$6@1bb9e06f -> wakeup thread pool-3-thread-32, errback=passthrough -> wakeup thread pool-3-thread-32) 00:27:41.081 [DEBUG - New I/O worker #1736] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.082 [DEBUG - New I/O worker #1894] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.082 [DEBUG - New I/O worker #1894] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1013), Traces: [0ms] refreshing cache from master, [621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [951ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [984ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1009ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:41.083 [DEBUG - New I/O worker #1894] (Deferred.java:1330) callback=retry RPC@952322921 returned Deferred@1023203497(state=PENDING, result=null, callback=(continuation of Deferred@2046706002 after retry RPC@952322921), errback=(continuation of Deferred@2046706002 after retry RPC@952322921)), so the following Deferred is getting paused: Deferred@2046706002(state=PAUSED, result=Deferred@1023203497, callback=org.apache.kudu.client.AsyncKuduClient$6@6a77bfcf -> wakeup thread pool-3-thread-34, errback=passthrough -> wakeup thread pool-3-thread-34) 00:27:41.085 [DEBUG - New I/O worker #1894] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.088 [DEBUG - New I/O worker #2085] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.024 [DEBUG - New I/O worker #1694] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.021 [DEBUG - New I/O worker #1923] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.021 [DEBUG - New I/O worker #2108] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.021 [DEBUG - New I/O worker #1930] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.021 [DEBUG - pool-3-thread-10] (Deferred.java:1330) callback=disconnect callback@816306098 returned Deferred@1376844972(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1811861879 after disconnect callback@816306098), errback=passthrough -> (continuation of Deferred@1811861879 after disconnect callback@816306098)), so the following Deferred is getting paused: Deferred@1811861879(state=PAUSED, result=Deferred@1376844972, callback=, errback=) 00:27:41.019 [DEBUG - New I/O worker #1744] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=776), Traces: [0ms] refreshing cache from master, [452ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [452ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [452ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [763ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:41.090 [DEBUG - New I/O worker #1744] (Deferred.java:1330) callback=retry RPC@1022021373 returned Deferred@953507256(state=PENDING, result=null, callback=(continuation of Deferred@2111292614 after retry RPC@1022021373), errback=(continuation of Deferred@2111292614 after retry RPC@1022021373)), so the following Deferred is getting paused: Deferred@2111292614(state=PAUSED, result=Deferred@953507256, callback=org.apache.kudu.client.AsyncKuduClient$6@261736c2 -> wakeup thread pool-3-thread-60, errback=passthrough -> wakeup thread pool-3-thread-60) 00:27:41.019 [DEBUG - New I/O worker #1739] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.015 [DEBUG - New I/O worker #1892] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.015 [DEBUG - New I/O worker #1898] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.091 [DEBUG - New I/O worker #1898] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.092 [DEBUG - New I/O worker #5] (Connection.java:429) [peer 1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)] encountered a read timeout; closing the channel 00:27:41.092 [DEBUG - pool-3-thread-57] (Deferred.java:1330) callback=disconnect callback@891627426 returned Deferred@842188842(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@818263919 after disconnect callback@891627426), errback=passthrough -> (continuation of Deferred@818263919 after disconnect callback@891627426)), so the following Deferred is getting paused: Deferred@818263919(state=PAUSED, result=Deferred@842188842, callback=, errback=) 00:27:41.092 [DEBUG - New I/O worker #1965] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.092 [DEBUG - New I/O worker #1966] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.015 [DEBUG - New I/O worker #1797] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.094 [DEBUG - New I/O worker #1744] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.092 [DEBUG - New I/O worker #5] (Connection.java:688) [peer 1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)] cleaning up while in state READY due to: [peer 1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)] encountered a read timeout; closing the channel 00:27:41.095 [DEBUG - New I/O worker #1898] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.095 [DEBUG - New I/O worker #1739] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.095 [DEBUG - New I/O worker #1739] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.095 [DEBUG - New I/O worker #1739] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.096 [DEBUG - New I/O worker #1739] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.096 [DEBUG - New I/O worker #1739] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.097 [DEBUG - pool-3-thread-18] (Deferred.java:1330) callback=disconnect callback@1727490263 returned Deferred@355493295(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1347789936 after disconnect callback@1727490263), errback=passthrough -> (continuation of Deferred@1347789936 after disconnect callback@1727490263)), so the following Deferred is getting paused: Deferred@1347789936(state=PAUSED, result=Deferred@355493295, callback=, errback=) 00:27:41.097 [DEBUG - New I/O worker #1742] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.097 [DEBUG - New I/O worker #2071] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.098 [DEBUG - New I/O worker #1892] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.098 [DEBUG - New I/O worker #1892] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.099 [DEBUG - New I/O worker #1892] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.099 [DEBUG - New I/O worker #1892] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.099 [DEBUG - New I/O worker #1892] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.100 [DEBUG - pool-3-thread-67] (Deferred.java:1330) callback=disconnect callback@864743496 returned Deferred@477492915(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@448923694 after disconnect callback@864743496), errback=passthrough -> (continuation of Deferred@448923694 after disconnect callback@864743496)), so the following Deferred is getting paused: Deferred@448923694(state=PAUSED, result=Deferred@477492915, callback=, errback=) 00:27:41.100 [DEBUG - New I/O worker #2163] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.101 [DEBUG - New I/O worker #1984] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.101 [DEBUG - New I/O worker #1736] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.101 [DEBUG - New I/O worker #1736] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.102 [DEBUG - New I/O worker #1736] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.102 [DEBUG - New I/O worker #1736] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.102 [DEBUG - New I/O worker #1744] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.102 [DEBUG - New I/O worker #1744] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.102 [DEBUG - New I/O worker #1744] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.102 [DEBUG - New I/O worker #1744] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.088 [DEBUG - New I/O worker #1894] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.084 [DEBUG - New I/O worker #1706] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.084 [DEBUG - New I/O worker #2086] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.084 [DEBUG - pool-3-thread-24] (Deferred.java:1330) callback=disconnect callback@1210478376 returned Deferred@2102766612(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2077668547 after disconnect callback@1210478376), errback=passthrough -> (continuation of Deferred@2077668547 after disconnect callback@1210478376)), so the following Deferred is getting paused: Deferred@2077668547(state=PAUSED, result=Deferred@2102766612, callback=, errback=) 00:27:41.083 [DEBUG - New I/O worker #1740] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=839), Traces: [0ms] refreshing cache from master, [453ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [454ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [454ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [739ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [756ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [798ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:41.110 [DEBUG - New I/O worker #1740] (Deferred.java:1330) callback=retry RPC@1550304192 returned Deferred@1482185876(state=PENDING, result=null, callback=(continuation of Deferred@492489211 after retry RPC@1550304192), errback=(continuation of Deferred@492489211 after retry RPC@1550304192)), so the following Deferred is getting paused: Deferred@492489211(state=PAUSED, result=Deferred@1482185876, callback=org.apache.kudu.client.AsyncKuduClient$6@4e16d2a3 -> wakeup thread pool-3-thread-19, errback=passthrough -> wakeup thread pool-3-thread-19) 00:27:41.080 [DEBUG - New I/O worker #1890] (Deferred.java:1330) callback=retry RPC@1493807933 returned Deferred@1563835588(state=PENDING, result=null, callback=(continuation of Deferred@406108422 after retry RPC@1493807933), errback=(continuation of Deferred@406108422 after retry RPC@1493807933)), so the following Deferred is getting paused: Deferred@406108422(state=PAUSED, result=Deferred@1563835588, callback=org.apache.kudu.client.AsyncKuduClient$6@3de23e43 -> wakeup thread pool-3-thread-73, errback=passthrough -> wakeup thread pool-3-thread-73) 00:27:41.075 [DEBUG - New I/O worker #1920] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.111 [DEBUG - New I/O worker #1920] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.111 [DEBUG - New I/O worker #1920] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.111 [DEBUG - New I/O worker #1920] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.111 [DEBUG - New I/O worker #1920] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.112 [DEBUG - New I/O worker #1927] (Negotiator.java:764) Authenticated connection [id: 0x0e3adbc7, /127.0.0.1:40688 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.112 [DEBUG - pool-3-thread-80] (Deferred.java:1330) callback=disconnect callback@1973973536 returned Deferred@140375142(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@143770651 after disconnect callback@1973973536), errback=passthrough -> (continuation of Deferred@143770651 after disconnect callback@1973973536)), so the following Deferred is getting paused: Deferred@143770651(state=PAUSED, result=Deferred@140375142, callback=, errback=) 00:27:41.112 [DEBUG - New I/O worker #2110] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.113 [DEBUG - New I/O worker #1920] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.075 [DEBUG - New I/O worker #1916] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.113 [DEBUG - New I/O worker #1916] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.114 [DEBUG - New I/O worker #1916] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.114 [DEBUG - New I/O worker #1916] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.114 [DEBUG - New I/O worker #1916] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.071 [DEBUG - New I/O worker #1806] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.115 [DEBUG - New I/O worker #1806] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.115 [DEBUG - pool-3-thread-64] (Deferred.java:1330) callback=disconnect callback@361075716 returned Deferred@1055579782(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@78738877 after disconnect callback@361075716), errback=passthrough -> (continuation of Deferred@78738877 after disconnect callback@361075716)), so the following Deferred is getting paused: Deferred@78738877(state=PAUSED, result=Deferred@1055579782, callback=, errback=) 00:27:41.068 [DEBUG - New I/O worker #1692] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.116 [DEBUG - New I/O worker #1931] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.068 [DEBUG - New I/O worker #2088] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.117 [DEBUG - New I/O worker #1740] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.118 [DEBUG - New I/O worker #1806] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299996, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.119 [DEBUG - New I/O worker #2058] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.119 [DEBUG - pool-3-thread-76] (Deferred.java:1330) callback=disconnect callback@748822160 returned Deferred@1485091124(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@61574657 after disconnect callback@748822160), errback=passthrough -> (continuation of Deferred@61574657 after disconnect callback@748822160)), so the following Deferred is getting paused: Deferred@61574657(state=PAUSED, result=Deferred@1485091124, callback=, errback=) 00:27:41.119 [DEBUG - New I/O worker #2060] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.068 [DEBUG - New I/O worker #1887] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=848), Traces: [0ms] refreshing cache from master, [484ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [484ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [485ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [722ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [796ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [849ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:41.120 [DEBUG - New I/O worker #1887] (Deferred.java:1330) callback=retry RPC@686780756 returned Deferred@751890577(state=PENDING, result=null, callback=(continuation of Deferred@1775401839 after retry RPC@686780756), errback=(continuation of Deferred@1775401839 after retry RPC@686780756)), so the following Deferred is getting paused: Deferred@1775401839(state=PAUSED, result=Deferred@751890577, callback=org.apache.kudu.client.AsyncKuduClient$6@bc71227 -> wakeup thread pool-3-thread-78, errback=passthrough -> wakeup thread pool-3-thread-78) 00:27:41.064 [DEBUG - New I/O worker #1675] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.061 [DEBUG - New I/O worker #1850] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.121 [DEBUG - New I/O worker #1850] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.121 [DEBUG - New I/O worker #1850] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.121 [DEBUG - New I/O worker #1850] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.122 [DEBUG - New I/O worker #1850] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.061 [DEBUG - New I/O worker #1926] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.120 [DEBUG - New I/O worker #1740] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.123 [DEBUG - New I/O worker #1740] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.123 [DEBUG - New I/O worker #1740] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.123 [DEBUG - New I/O worker #1740] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.124 [DEBUG - New I/O worker #1740] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.124 [DEBUG - pool-3-thread-19] (Deferred.java:1330) callback=disconnect callback@797375373 returned Deferred@1587617900(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@103136509 after disconnect callback@797375373), errback=passthrough -> (continuation of Deferred@103136509 after disconnect callback@797375373)), so the following Deferred is getting paused: Deferred@103136509(state=PAUSED, result=Deferred@1587617900, callback=, errback=) 00:27:41.125 [DEBUG - New I/O worker #2090] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.125 [DEBUG - New I/O worker #1887] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.126 [DEBUG - pool-3-thread-23] (Deferred.java:1330) callback=disconnect callback@1402135130 returned Deferred@1763388486(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1709731170 after disconnect callback@1402135130), errback=passthrough -> (continuation of Deferred@1709731170 after disconnect callback@1402135130)), so the following Deferred is getting paused: Deferred@1709731170(state=PAUSED, result=Deferred@1763388486, callback=, errback=) 00:27:41.126 [DEBUG - New I/O worker #2027] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.126 [DEBUG - New I/O worker #2028] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.127 [DEBUG - New I/O worker #1916] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.128 [DEBUG - New I/O worker #1887] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.128 [DEBUG - New I/O worker #1887] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.128 [DEBUG - New I/O worker #1887] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.129 [DEBUG - New I/O worker #1887] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.119 [DEBUG - New I/O worker #1806] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.111 [DEBUG - New I/O worker #1744] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299994, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299994, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299994, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.111 [DEBUG - New I/O worker #1890] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.130 [DEBUG - pool-3-thread-60] (Deferred.java:1330) callback=disconnect callback@923859418 returned Deferred@13515075(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@746580528 after disconnect callback@923859418), errback=passthrough -> (continuation of Deferred@746580528 after disconnect callback@923859418)), so the following Deferred is getting paused: Deferred@746580528(state=PAUSED, result=Deferred@13515075, callback=, errback=) 00:27:41.130 [DEBUG - New I/O worker #2109] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.131 [DEBUG - New I/O worker #2138] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.132 [DEBUG - New I/O worker #1932] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.135 [DEBUG - New I/O worker #1675] (Negotiator.java:764) Authenticated connection [id: 0x920143fb, /127.0.0.1:38708 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.135 [DEBUG - New I/O worker #1692] (Negotiator.java:764) Authenticated connection [id: 0xd0aee070, /127.0.0.1:38706 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.136 [DEBUG - New I/O worker #1744] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.108 [DEBUG - New I/O worker #1894] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.137 [DEBUG - New I/O worker #1894] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.137 [DEBUG - New I/O worker #1894] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.137 [DEBUG - New I/O worker #1894] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.103 [DEBUG - New I/O worker #1892] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.102 [DEBUG - New I/O worker #1736] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.138 [DEBUG - pool-3-thread-34] (Deferred.java:1330) callback=disconnect callback@1181105373 returned Deferred@638959488(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@584590038 after disconnect callback@1181105373), errback=passthrough -> (continuation of Deferred@584590038 after disconnect callback@1181105373)), so the following Deferred is getting paused: Deferred@584590038(state=PAUSED, result=Deferred@638959488, callback=, errback=) 00:27:41.138 [DEBUG - New I/O worker #1978] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.139 [DEBUG - New I/O worker #2074] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.139 [DEBUG - New I/O worker #1736] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.142 [DEBUG - New I/O worker #1887] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.143 [DEBUG - pool-3-thread-78] (Deferred.java:1330) callback=disconnect callback@96890790 returned Deferred@614469251(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1997317405 after disconnect callback@96890790), errback=passthrough -> (continuation of Deferred@1997317405 after disconnect callback@96890790)), so the following Deferred is getting paused: Deferred@1997317405(state=PAUSED, result=Deferred@614469251, callback=, errback=) 00:27:41.143 [DEBUG - New I/O worker #1995] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.143 [DEBUG - New I/O worker #1994] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.144 [DEBUG - New I/O worker #1887] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.144 [DEBUG - New I/O worker #2088] (Negotiator.java:764) Authenticated connection [id: 0x4204e4f3, /127.0.0.1:54054 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.145 [DEBUG - New I/O worker #2073] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.102 [DEBUG - New I/O worker #2089] (Negotiator.java:764) Authenticated connection [id: 0xb629c01f, /127.0.0.1:40684 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.098 [DEBUG - New I/O worker #1739] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.095 [DEBUG - New I/O worker #1694] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=556), Traces: [0ms] refreshing cache from master, [219ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [220ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [485ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [486ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [489ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:41.094 [DEBUG - New I/O worker #1797] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=854), Traces: [0ms] refreshing cache from master, [459ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [459ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [459ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [768ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [771ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [823ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:41.152 [DEBUG - New I/O worker #1797] (Deferred.java:1330) callback=retry RPC@327270296 returned Deferred@398338352(state=PENDING, result=null, callback=(continuation of Deferred@1388098979 after retry RPC@327270296), errback=(continuation of Deferred@1388098979 after retry RPC@327270296)), so the following Deferred is getting paused: Deferred@1388098979(state=PAUSED, result=Deferred@398338352, callback=org.apache.kudu.client.AsyncKuduClient$6@7dc7b9f6 -> wakeup thread pool-3-thread-63, errback=passthrough -> wakeup thread pool-3-thread-63) 00:27:41.153 [DEBUG - New I/O worker #1926] (Negotiator.java:764) Authenticated connection [id: 0xd2aeed2c, /127.0.0.1:54058 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.155 [DEBUG - New I/O worker #1797] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.157 [DEBUG - New I/O worker #1890] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.157 [DEBUG - New I/O worker #1890] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.157 [DEBUG - New I/O worker #1890] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.157 [DEBUG - New I/O worker #1890] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.158 [DEBUG - New I/O worker #1890] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.158 [DEBUG - pool-3-thread-73] (Deferred.java:1330) callback=disconnect callback@682745016 returned Deferred@1072592715(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1676939981 after disconnect callback@682745016), errback=passthrough -> (continuation of Deferred@1676939981 after disconnect callback@682745016)), so the following Deferred is getting paused: Deferred@1676939981(state=PAUSED, result=Deferred@1072592715, callback=, errback=) 00:27:41.158 [DEBUG - New I/O worker #2157] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.159 [DEBUG - New I/O worker #1890] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.152 [DEBUG - New I/O worker #1694] (Deferred.java:1330) callback=retry RPC@379995381 returned Deferred@312070905(state=PENDING, result=null, callback=(continuation of Deferred@1469796046 after retry RPC@379995381), errback=(continuation of Deferred@1469796046 after retry RPC@379995381)), so the following Deferred is getting paused: Deferred@1469796046(state=PAUSED, result=Deferred@312070905, callback=org.apache.kudu.client.AsyncKuduClient$6@42230845 -> wakeup thread pool-3-thread-70, errback=passthrough -> wakeup thread pool-3-thread-70) 00:27:41.139 [DEBUG - New I/O worker #1894] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.139 [DEBUG - New I/O worker #1979] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.138 [DEBUG - pool-3-thread-32] (Deferred.java:1330) callback=disconnect callback@654028111 returned Deferred@690900152(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@230314736 after disconnect callback@654028111), errback=passthrough -> (continuation of Deferred@230314736 after disconnect callback@654028111)), so the following Deferred is getting paused: Deferred@230314736(state=PAUSED, result=Deferred@690900152, callback=, errback=) 00:27:41.137 [DEBUG - New I/O worker #1692] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.131 [DEBUG - New I/O worker #2070] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.129 [DEBUG - New I/O worker #1850] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.129 [DEBUG - New I/O worker #1740] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.125 [DEBUG - New I/O worker #2136] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.161 [DEBUG - New I/O worker #1985] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.162 [DEBUG - New I/O worker #1692] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=273), Traces: [0ms] refreshing cache from master, [150ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [151ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [153ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [248ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [258ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [264ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:41.163 [DEBUG - New I/O worker #1675] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.163 [DEBUG - New I/O worker #1675] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=266), Traces: [0ms] refreshing cache from master, [145ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [149ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [153ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [248ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [259ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [266ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:41.164 [DEBUG - New I/O worker #1692] (Deferred.java:1330) callback=retry RPC@1193845625 returned Deferred@1125596996(state=PENDING, result=null, callback=(continuation of Deferred@102082882 after retry RPC@1193845625), errback=(continuation of Deferred@102082882 after retry RPC@1193845625)), so the following Deferred is getting paused: Deferred@102082882(state=PAUSED, result=Deferred@1125596996, callback=org.apache.kudu.client.AsyncKuduClient$6@35680340 -> wakeup thread pool-3-thread-15, errback=passthrough -> wakeup thread pool-3-thread-15) 00:27:41.165 [DEBUG - New I/O worker #1692] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.167 [DEBUG - New I/O worker #1675] (Deferred.java:1330) callback=retry RPC@372431958 returned Deferred@302864968(state=PENDING, result=null, callback=(continuation of Deferred@1460655725 after retry RPC@372431958), errback=(continuation of Deferred@1460655725 after retry RPC@372431958)), so the following Deferred is getting paused: Deferred@1460655725(state=PAUSED, result=Deferred@302864968, callback=org.apache.kudu.client.AsyncKuduClient$6@66425145 -> wakeup thread pool-3-thread-14, errback=passthrough -> wakeup thread pool-3-thread-14) 00:27:41.167 [DEBUG - New I/O worker #1675] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.170 [DEBUG - New I/O worker #1692] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.170 [DEBUG - New I/O worker #1692] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.170 [DEBUG - New I/O worker #1675] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.170 [DEBUG - New I/O worker #1675] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.171 [DEBUG - New I/O worker #1675] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.171 [DEBUG - New I/O worker #1675] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.174 [DEBUG - New I/O worker #1675] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.170 [DEBUG - New I/O worker #1692] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.174 [DEBUG - New I/O worker #1694] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.173 [DEBUG - New I/O worker #1797] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.176 [DEBUG - New I/O worker #1797] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.176 [DEBUG - New I/O worker #1797] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.176 [DEBUG - New I/O worker #1797] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.176 [DEBUG - New I/O worker #1797] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.177 [DEBUG - New I/O worker #1692] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.177 [DEBUG - pool-3-thread-63] (Deferred.java:1330) callback=disconnect callback@1885569593 returned Deferred@1815423617(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@115759407 after disconnect callback@1885569593), errback=passthrough -> (continuation of Deferred@115759407 after disconnect callback@1885569593)), so the following Deferred is getting paused: Deferred@115759407(state=PAUSED, result=Deferred@1815423617, callback=, errback=) 00:27:41.177 [DEBUG - New I/O worker #2067] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.177 [DEBUG - New I/O worker #2091] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.178 [DEBUG - New I/O worker #1692] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.178 [DEBUG - New I/O worker #1797] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.179 [DEBUG - New I/O worker #2088] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.180 [DEBUG - pool-3-thread-15] (Deferred.java:1330) callback=disconnect callback@1029445609 returned Deferred@355299976(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1827554227 after disconnect callback@1029445609), errback=passthrough -> (continuation of Deferred@1827554227 after disconnect callback@1029445609)), so the following Deferred is getting paused: Deferred@1827554227(state=PAUSED, result=Deferred@355299976, callback=, errback=) 00:27:41.180 [DEBUG - New I/O worker #2089] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.184 [DEBUG - New I/O worker #1692] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.186 [DEBUG - pool-3-thread-14] (Deferred.java:1330) callback=disconnect callback@1854889006 returned Deferred@1503501071(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@254951131 after disconnect callback@1854889006), errback=passthrough -> (continuation of Deferred@254951131 after disconnect callback@1854889006)), so the following Deferred is getting paused: Deferred@254951131(state=PAUSED, result=Deferred@1503501071, callback=, errback=) 00:27:41.186 [DEBUG - New I/O worker #1926] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.187 [DEBUG - New I/O worker #1927] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.186 [DEBUG - New I/O worker #1694] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.187 [DEBUG - New I/O worker #1694] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.187 [DEBUG - New I/O worker #1694] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.188 [DEBUG - New I/O worker #1694] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.188 [DEBUG - New I/O worker #1692] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.188 [DEBUG - New I/O worker #1694] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.188 [DEBUG - New I/O worker #1675] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.189 [DEBUG - pool-3-thread-70] (Deferred.java:1330) callback=disconnect callback@1307423061 returned Deferred@1071396801(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@933040730 after disconnect callback@1307423061), errback=passthrough -> (continuation of Deferred@933040730 after disconnect callback@1307423061)), so the following Deferred is getting paused: Deferred@933040730(state=PAUSED, result=Deferred@1071396801, callback=, errback=) 00:27:41.189 [DEBUG - New I/O worker #1696] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.190 [DEBUG - New I/O worker #1694] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.191 [DEBUG - New I/O worker #1695] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.215 [DEBUG - pool-3-thread-74] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.217 [DEBUG - pool-3-thread-44] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.218 [DEBUG - pool-3-thread-12] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.220 [DEBUG - pool-3-thread-75] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.222 [DEBUG - pool-3-thread-26] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.236 [DEBUG - New I/O worker #1797] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.237 [DEBUG - New I/O worker #1985] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.237 [DEBUG - New I/O worker #1740] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.239 [DEBUG - pool-3-thread-41] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.243 [DEBUG - New I/O worker #1850] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.243 [DEBUG - New I/O worker #1894] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.247 [DEBUG - New I/O worker #1739] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.251 [DEBUG - New I/O worker #2073] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.252 [DEBUG - pool-3-thread-2] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.254 [DEBUG - New I/O worker #1887] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.254 [DEBUG - New I/O worker #1892] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.255 [DEBUG - New I/O worker #1744] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.263 [DEBUG - New I/O worker #1932] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.267 [DEBUG - New I/O worker #1806] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.339 [DEBUG - New I/O worker #1898] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.343 [DEBUG - New I/O worker #1923] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.343 [DEBUG - New I/O worker #2085] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.388 [DEBUG - pool-3-thread-29] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.412 [DEBUG - pool-3-thread-4] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.419 [DEBUG - New I/O worker #2109] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.424 [DEBUG - pool-3-thread-13] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.428 [DEBUG - New I/O worker #2296] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.428 [DEBUG - New I/O worker #2302] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.429 [DEBUG - New I/O worker #2360] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.430 [DEBUG - New I/O worker #2359] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.431 [DEBUG - New I/O worker #2369] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.432 [DEBUG - New I/O worker #2368] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.458 [DEBUG - New I/O worker #2359] (Negotiator.java:764) Authenticated connection [id: 0x4c1c514a, /127.0.0.1:54064 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.463 [DEBUG - New I/O worker #2296] (Negotiator.java:764) Authenticated connection [id: 0x73aab2a6, /127.0.0.1:38718 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.465 [DEBUG - New I/O worker #2296] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.466 [DEBUG - New I/O worker #2296] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=213), Traces: [0ms] refreshing cache from master, [171ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [172ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [172ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [207ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [212ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:41.466 [DEBUG - New I/O worker #2296] (Deferred.java:1330) callback=retry RPC@1381504065 returned Deferred@1449626322(state=PENDING, result=null, callback=(continuation of Deferred@325394042 after retry RPC@1381504065), errback=(continuation of Deferred@325394042 after retry RPC@1381504065)), so the following Deferred is getting paused: Deferred@325394042(state=PAUSED, result=Deferred@1449626322, callback=org.apache.kudu.client.AsyncKuduClient$6@30546f45 -> wakeup thread pool-3-thread-2, errback=passthrough -> wakeup thread pool-3-thread-2) 00:27:41.468 [DEBUG - New I/O worker #2296] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.468 [DEBUG - New I/O worker #2360] (Negotiator.java:764) Authenticated connection [id: 0x4f25b8cc, /127.0.0.1:40694 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.468 [DEBUG - New I/O worker #2369] (Negotiator.java:764) Authenticated connection [id: 0x64673a8f, /127.0.0.1:40700 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.471 [DEBUG - New I/O worker #2296] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.472 [DEBUG - New I/O worker #2296] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.472 [DEBUG - New I/O worker #2296] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.472 [DEBUG - New I/O worker #2296] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.473 [DEBUG - New I/O worker #2296] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.473 [DEBUG - New I/O worker #2296] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.473 [DEBUG - New I/O worker #2360] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.473 [DEBUG - pool-3-thread-2] (Deferred.java:1330) callback=disconnect callback@1423605389 returned Deferred@2049414382(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1779905942 after disconnect callback@1423605389), errback=passthrough -> (continuation of Deferred@1779905942 after disconnect callback@1423605389)), so the following Deferred is getting paused: Deferred@1779905942(state=PAUSED, result=Deferred@2049414382, callback=, errback=) 00:27:41.478 [DEBUG - New I/O worker #2359] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.478 [DEBUG - New I/O worker #2368] (Negotiator.java:764) Authenticated connection [id: 0x07d30da9, /127.0.0.1:54070 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.481 [DEBUG - New I/O worker #2302] (Negotiator.java:764) Authenticated connection [id: 0x42ae4175, /127.0.0.1:38724 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.484 [DEBUG - New I/O worker #2302] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.484 [DEBUG - New I/O worker #2302] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=244), Traces: [0ms] refreshing cache from master, [186ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [186ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [186ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [242ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [242ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [244ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:41.484 [DEBUG - New I/O worker #2302] (Deferred.java:1330) callback=retry RPC@366265803 returned Deferred@300630953(state=PENDING, result=null, callback=(continuation of Deferred@1424609264 after retry RPC@366265803), errback=(continuation of Deferred@1424609264 after retry RPC@366265803)), so the following Deferred is getting paused: Deferred@1424609264(state=PAUSED, result=Deferred@300630953, callback=org.apache.kudu.client.AsyncKuduClient$6@56be7054 -> wakeup thread pool-3-thread-41, errback=passthrough -> wakeup thread pool-3-thread-41) 00:27:41.488 [DEBUG - New I/O worker #2280] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.488 [DEBUG - New I/O worker #2380] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.490 [DEBUG - New I/O worker #2395] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.492 [DEBUG - New I/O worker #2282] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.495 [DEBUG - New I/O worker #2379] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.501 [DEBUG - New I/O worker #2394] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.505 [DEBUG - New I/O worker #1675] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.515 [DEBUG - New I/O worker #2380] (Negotiator.java:764) Authenticated connection [id: 0xdeb1dec9, /127.0.0.1:40706 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.518 [DEBUG - New I/O worker #2302] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.519 [DEBUG - New I/O worker #1695] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.523 [DEBUG - New I/O worker #2284] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.531 [DEBUG - New I/O worker #2282] (Negotiator.java:764) Authenticated connection [id: 0x2e859a7f, /127.0.0.1:38736 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.536 [DEBUG - New I/O worker #2395] (Negotiator.java:764) Authenticated connection [id: 0x160bd11d, /127.0.0.1:40712 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.536 [DEBUG - New I/O worker #2405] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.535 [DEBUG - New I/O worker #2379] (Negotiator.java:764) Authenticated connection [id: 0xd041ef0a, /127.0.0.1:54076 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.542 [DEBUG - New I/O worker #2282] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.542 [DEBUG - New I/O worker #2282] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=322), Traces: [0ms] refreshing cache from master, [268ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [268ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [268ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [322ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:41.543 [DEBUG - New I/O worker #2284] (Negotiator.java:764) Authenticated connection [id: 0x6f155c5e, /127.0.0.1:38742 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.545 [DEBUG - New I/O worker #2282] (Deferred.java:1330) callback=retry RPC@1984314062 returned Deferred@1920577758(state=PENDING, result=null, callback=(continuation of Deferred@930820341 after retry RPC@1984314062), errback=(continuation of Deferred@930820341 after retry RPC@1984314062)), so the following Deferred is getting paused: Deferred@930820341(state=PAUSED, result=Deferred@1920577758, callback=org.apache.kudu.client.AsyncKuduClient$6@41d1c11f -> wakeup thread pool-3-thread-75, errback=passthrough -> wakeup thread pool-3-thread-75) 00:27:41.545 [DEBUG - New I/O worker #2282] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.549 [DEBUG - New I/O worker #2284] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.549 [DEBUG - New I/O worker #2284] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=330), Traces: [0ms] refreshing cache from master, [300ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [330ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:41.553 [DEBUG - New I/O worker #2394] (Negotiator.java:764) Authenticated connection [id: 0xe6f4b63d, /127.0.0.1:54082 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.555 [DEBUG - New I/O worker #2280] (Negotiator.java:764) Authenticated connection [id: 0xab15115f, /127.0.0.1:38730 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.556 [DEBUG - New I/O worker #2282] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.557 [DEBUG - New I/O worker #2282] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.557 [DEBUG - New I/O worker #2282] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.557 [DEBUG - New I/O worker #2282] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.557 [DEBUG - New I/O worker #2282] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.558 [DEBUG - pool-3-thread-75] (Deferred.java:1330) callback=disconnect callback@366711976 returned Deferred@9722914(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1205333153 after disconnect callback@366711976), errback=passthrough -> (continuation of Deferred@1205333153 after disconnect callback@366711976)), so the following Deferred is getting paused: Deferred@1205333153(state=PAUSED, result=Deferred@9722914, callback=, errback=) 00:27:41.558 [DEBUG - New I/O worker #2394] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.559 [INFO - New I/O worker #2394] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:41.559 [DEBUG - New I/O worker #2395] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.559 [INFO - New I/O worker #2395] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:41.560 [DEBUG - New I/O worker #2282] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:41.564676 22624 connection.cc:554] Connection torn down before Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:54082 (request call id 0) could send its response 00:27:41.565 [DEBUG - New I/O worker #2407] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.567 [DEBUG - New I/O worker #2405] (Negotiator.java:764) Authenticated connection [id: 0xdb0f927b, /127.0.0.1:54088 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:41.570565 22688 connection.cc:320] Error closing socket: Network error: TlsSocket::Close: Connection reset by peer 00:27:41.570 [DEBUG - New I/O worker #2284] (Deferred.java:1330) callback=retry RPC@953236667 returned Deferred@1022279545(state=PENDING, result=null, callback=(continuation of Deferred@2045523584 after retry RPC@953236667), errback=(continuation of Deferred@2045523584 after retry RPC@953236667)), so the following Deferred is getting paused: Deferred@2045523584(state=PAUSED, result=Deferred@1022279545, callback=org.apache.kudu.client.AsyncKuduClient$6@795a1328 -> wakeup thread pool-3-thread-12, errback=passthrough -> wakeup thread pool-3-thread-12) 00:27:41.571 [DEBUG - New I/O worker #2410] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.572 [DEBUG - New I/O worker #2284] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.557 [DEBUG - New I/O worker #2280] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.576 [DEBUG - New I/O worker #2302] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.576 [DEBUG - New I/O worker #2302] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.583 [DEBUG - New I/O worker #2302] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.583 [DEBUG - New I/O worker #2302] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.561 [DEBUG - New I/O worker #2408] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.561 [DEBUG - New I/O worker #2273] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.586 [DEBUG - New I/O worker #2410] (Negotiator.java:764) Authenticated connection [id: 0x3d7c1278, /127.0.0.1:40724 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.584 [DEBUG - New I/O worker #2280] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=362), Traces: [0ms] refreshing cache from master, [264ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [264ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [265ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [318ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [335ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [347ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:41.583 [DEBUG - New I/O worker #2284] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.581 [DEBUG - pool-3-thread-69] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.581 [DEBUG - pool-3-thread-46] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.580 [DEBUG - New I/O worker #2288] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.578 [DEBUG - New I/O worker #2419] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.600 [DEBUG - New I/O worker #2418] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.611 [DEBUG - New I/O worker #2273] (Negotiator.java:764) Authenticated connection [id: 0x3fa35d05, /127.0.0.1:38748 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.618 [DEBUG - New I/O worker #2408] (Negotiator.java:764) Authenticated connection [id: 0x7721948a, /127.0.0.1:54094 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.624 [DEBUG - New I/O worker #2284] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.628 [DEBUG - New I/O worker #2280] (Deferred.java:1330) callback=retry RPC@1697204398 returned Deferred@1628885203(state=PENDING, result=null, callback=(continuation of Deferred@605307541 after retry RPC@1697204398), errback=(continuation of Deferred@605307541 after retry RPC@1697204398)), so the following Deferred is getting paused: Deferred@605307541(state=PAUSED, result=Deferred@1628885203, callback=org.apache.kudu.client.AsyncKuduClient$6@3a39c617 -> wakeup thread pool-3-thread-26, errback=passthrough -> wakeup thread pool-3-thread-26) 00:27:41.633 [DEBUG - New I/O worker #2288] (Negotiator.java:764) Authenticated connection [id: 0x2bcc75bd, /127.0.0.1:38754 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.629 [DEBUG - New I/O worker #2302] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299982, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299963, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299962, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.629 [DEBUG - New I/O worker #2273] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.644 [DEBUG - pool-3-thread-5] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.644 [DEBUG - New I/O worker #2273] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=426), Traces: [0ms] refreshing cache from master, [341ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [342ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [343ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [383ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [403ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [411ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:41.645 [DEBUG - New I/O worker #2273] (Deferred.java:1330) callback=retry RPC@1881498877 returned Deferred@1947926049(state=PENDING, result=null, callback=(continuation of Deferred@823684806 after retry RPC@1881498877), errback=(continuation of Deferred@823684806 after retry RPC@1881498877)), so the following Deferred is getting paused: Deferred@823684806(state=PAUSED, result=Deferred@1947926049, callback=org.apache.kudu.client.AsyncKuduClient$6@5b3a2da9 -> wakeup thread pool-3-thread-44, errback=passthrough -> wakeup thread pool-3-thread-44) 00:27:41.645 [DEBUG - New I/O worker #2369] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.645 [DEBUG - New I/O worker #2302] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.646 [DEBUG - pool-3-thread-22] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.646 [DEBUG - pool-3-thread-33] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.629 [DEBUG - New I/O worker #2284] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.628 [DEBUG - pool-3-thread-37] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.647 [DEBUG - New I/O worker #2284] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.648 [DEBUG - New I/O worker #2284] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.648 [DEBUG - New I/O worker #2273] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.648 [DEBUG - pool-3-thread-12] (Deferred.java:1330) callback=disconnect callback@1108872783 returned Deferred@67454098(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2085669826 after disconnect callback@1108872783), errback=passthrough -> (continuation of Deferred@2085669826 after disconnect callback@1108872783)), so the following Deferred is getting paused: Deferred@2085669826(state=PAUSED, result=Deferred@67454098, callback=, errback=) 00:27:41.628 [DEBUG - New I/O worker #2407] (Negotiator.java:764) Authenticated connection [id: 0x32287aaf, /127.0.0.1:40718 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.649 [DEBUG - New I/O worker #2405] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.649 [DEBUG - New I/O worker #2407] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.649 [INFO - New I/O worker #2407] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:41.645 [DEBUG - New I/O worker #2368] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.645 [DEBUG - pool-3-thread-41] (Deferred.java:1330) callback=disconnect callback@1395378292 returned Deferred@989352037(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@637626559 after disconnect callback@1395378292), errback=passthrough -> (continuation of Deferred@637626559 after disconnect callback@1395378292)), so the following Deferred is getting paused: Deferred@637626559(state=PAUSED, result=Deferred@989352037, callback=, errback=) 00:27:41.643 [DEBUG - New I/O worker #2418] (Negotiator.java:764) Authenticated connection [id: 0x368550e3, /127.0.0.1:54100 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.643 [DEBUG - pool-3-thread-3] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.650 [DEBUG - New I/O worker #2284] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.639 [DEBUG - New I/O worker #2288] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.650 [DEBUG - New I/O worker #2366] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.651 [DEBUG - New I/O worker #2288] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=435), Traces: [0ms] refreshing cache from master, [361ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [361ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [361ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [424ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:41.653 [DEBUG - New I/O worker #2299] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.654 [DEBUG - New I/O worker #2288] (Deferred.java:1330) callback=retry RPC@1385937657 returned Deferred@1453597810(state=PENDING, result=null, callback=(continuation of Deferred@329691330 after retry RPC@1385937657), errback=(continuation of Deferred@329691330 after retry RPC@1385937657)), so the following Deferred is getting paused: Deferred@329691330(state=PAUSED, result=Deferred@1453597810, callback=org.apache.kudu.client.AsyncKuduClient$6@2d3cd9eb -> wakeup thread pool-3-thread-74, errback=passthrough -> wakeup thread pool-3-thread-74) 00:27:41.655 [DEBUG - New I/O worker #2288] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.638 [DEBUG - New I/O worker #2280] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:41.638 [DEBUG - New I/O worker #2419] (Negotiator.java:764) Authenticated connection [id: 0x4d910e9d, /127.0.0.1:40730 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.668 [DEBUG - New I/O worker #2279] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.671 [DEBUG - New I/O worker #2288] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.671 [DEBUG - New I/O worker #2288] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.660 [DEBUG - New I/O worker #2273] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.649 [DEBUG - New I/O worker #2292] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.674 [DEBUG - New I/O worker #2288] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.674 [DEBUG - New I/O worker #2250] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.673 [DEBUG - New I/O worker #2420] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.688 [DEBUG - New I/O worker #2273] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.699 [DEBUG - New I/O worker #2288] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.701 [DEBUG - New I/O worker #2280] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:41.701 [DEBUG - New I/O worker #2280] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:41.701 [DEBUG - New I/O worker #2280] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.702 [DEBUG - New I/O worker #2280] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.702 [DEBUG - New I/O worker #2280] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.703 [DEBUG - New I/O worker #2353] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.703 [DEBUG - New I/O worker #2380] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.703 [DEBUG - New I/O worker #2379] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.704 [DEBUG - New I/O worker #2273] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:41.704 [DEBUG - pool-3-thread-26] (Deferred.java:1330) callback=disconnect callback@301782259 returned Deferred@1595821643(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1536407576 after disconnect callback@301782259), errback=passthrough -> (continuation of Deferred@1536407576 after disconnect callback@301782259)), so the following Deferred is getting paused: Deferred@1536407576(state=PAUSED, result=Deferred@1595821643, callback=, errback=) 00:27:41.705 [DEBUG - New I/O worker #2280] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.709 [DEBUG - New I/O worker #2273] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:41.710 [DEBUG - New I/O worker #2366] (Negotiator.java:764) Authenticated connection [id: 0x129b93e7, /127.0.0.1:40736 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.715 [DEBUG - New I/O worker #2251] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.711 [DEBUG - New I/O worker #2290] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.711 [DEBUG - New I/O worker #2269] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.723 [DEBUG - New I/O worker #2302] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.711 [DEBUG - New I/O worker #2286] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.732 [DEBUG - New I/O worker #2282] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.728 [DEBUG - New I/O worker #2359] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.728 [DEBUG - pool-3-thread-53] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.720 [DEBUG - New I/O worker #2327] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.750 [DEBUG - New I/O worker #2288] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299989, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299980, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299980, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.743 [DEBUG - New I/O worker #2284] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.787 [DEBUG - pool-3-thread-25] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.735 [DEBUG - New I/O worker #2299] (Negotiator.java:764) Authenticated connection [id: 0x10e0a38d, /127.0.0.1:54106 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.795 [DEBUG - New I/O worker #2426] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.793 [DEBUG - New I/O worker #2386] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.788 [DEBUG - New I/O worker #2419] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.788 [DEBUG - pool-3-thread-74] (Deferred.java:1330) callback=disconnect callback@1250541608 returned Deferred@1422568221(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@978402526 after disconnect callback@1250541608), errback=passthrough -> (continuation of Deferred@978402526 after disconnect callback@1250541608)), so the following Deferred is getting paused: Deferred@978402526(state=PAUSED, result=Deferred@1422568221, callback=, errback=) 00:27:41.788 [DEBUG - New I/O worker #2418] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.788 [DEBUG - New I/O worker #2280] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.787 [DEBUG - New I/O worker #2288] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.787 [DEBUG - New I/O worker #2278] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.787 [DEBUG - New I/O worker #2276] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.786 [DEBUG - New I/O worker #2279] (Negotiator.java:764) Authenticated connection [id: 0x9c37e266, /127.0.0.1:54112 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.786 [DEBUG - New I/O worker #2313] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.786 [DEBUG - New I/O worker #2319] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.785 [DEBUG - New I/O worker #2423] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.785 [DEBUG - New I/O worker #2424] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.785 [DEBUG - New I/O worker #2430] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.780 [DEBUG - New I/O worker #2317] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.774 [DEBUG - pool-3-thread-36] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.773 [DEBUG - New I/O worker #2315] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.773 [DEBUG - New I/O worker #2393] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.773 [DEBUG - New I/O worker #2245] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.773 [DEBUG - New I/O worker #2277] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.773 [DEBUG - New I/O worker #2377] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.772 [DEBUG - New I/O worker #2461] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.772 [DEBUG - New I/O worker #2390] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.772 [DEBUG - New I/O worker #2375] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.923 [DEBUG - New I/O worker #2292] (Negotiator.java:764) Authenticated connection [id: 0x41364840, /127.0.0.1:38760 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.770 [DEBUG - New I/O worker #2273] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299993, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299987, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299978, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:41.925 [DEBUG - pool-3-thread-44] (Deferred.java:1330) callback=disconnect callback@1023006484 returned Deferred@510511054(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1895238679 after disconnect callback@1023006484), errback=passthrough -> (continuation of Deferred@1895238679 after disconnect callback@1023006484)), so the following Deferred is getting paused: Deferred@1895238679(state=PAUSED, result=Deferred@510511054, callback=, errback=) 00:27:41.925 [DEBUG - New I/O worker #2410] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.927 [DEBUG - New I/O worker #2408] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:41.931 [DEBUG - New I/O worker #2273] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:41.921 [DEBUG - New I/O worker #2353] (Negotiator.java:764) Authenticated connection [id: 0xfebc8330, /127.0.0.1:40750 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.914 [DEBUG - New I/O worker #2256] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.912 [DEBUG - New I/O worker #2331] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.908 [DEBUG - New I/O worker #2260] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.905 [DEBUG - New I/O worker #2325] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:41.900 [DEBUG - pool-3-thread-59] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.899 [DEBUG - pool-3-thread-47] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.897 [DEBUG - New I/O worker #2503] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:41.895 [DEBUG - New I/O worker #2322] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:41.856 [DEBUG - pool-3-thread-48] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.836 [DEBUG - New I/O worker #2419] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:41.831 [DEBUG - pool-3-thread-9] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.825 [DEBUG - New I/O worker #2290] (Negotiator.java:764) Authenticated connection [id: 0x5e5a36fa, /127.0.0.1:54118 => /127.22.19.254:44849] using SASL/PLAIN 00:27:41.821 [DEBUG - New I/O worker #2250] (Negotiator.java:764) Authenticated connection [id: 0x702e2784, /127.0.0.1:38766 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.819 [DEBUG - New I/O worker #2327] (Negotiator.java:764) Authenticated connection [id: 0x079256a5, /127.0.0.1:40754 => /127.22.19.253:37879] using SASL/PLAIN 00:27:41.816 [DEBUG - pool-3-thread-6] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.807 [DEBUG - pool-3-thread-27] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.802 [DEBUG - pool-3-thread-52] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.028 [DEBUG - New I/O worker #2424] (Negotiator.java:764) Authenticated connection [id: 0x6a578450, /127.0.0.1:40790 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.026 [DEBUG - New I/O worker #2276] (Negotiator.java:764) Authenticated connection [id: 0x4aeb64f7, /127.0.0.1:38802 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.019 [DEBUG - New I/O worker #2286] (Negotiator.java:764) Authenticated connection [id: 0x1a6eeb3e, /127.0.0.1:38772 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.018 [DEBUG - New I/O worker #2315] (Negotiator.java:764) Authenticated connection [id: 0xd2232d0a, /127.0.0.1:38788 => /127.22.19.252:32833] using SASL/PLAIN 00:27:41.989 [DEBUG - New I/O worker #2292] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:41.989 [DEBUG - pool-3-thread-38] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.989 [DEBUG - pool-3-thread-36] (NetUtil.java:106) Resolved IP of `127.22.19.254' to /127.22.19.254 in 6567867ns 00:27:41.975 [DEBUG - New I/O worker #2273] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:41.957 [DEBUG - pool-3-thread-30] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:41.947 [DEBUG - New I/O worker #2269] (Negotiator.java:764) Authenticated connection [id: 0x12f6b2db, /127.0.0.1:54124 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.062 [DEBUG - pool-3-thread-52] (NetUtil.java:106) Resolved IP of `127.22.19.254' to /127.22.19.254 in 8939926ns 00:27:42.060 [DEBUG - New I/O worker #2292] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=633), Traces: [0ms] refreshing cache from master, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [223ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [224ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [307ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [374ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [537ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.053 [DEBUG - New I/O worker #2277] (Negotiator.java:764) Authenticated connection [id: 0x88da428f, /127.0.0.1:54150 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.053 [DEBUG - pool-3-thread-21] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.043 [DEBUG - New I/O worker #2430] (Negotiator.java:764) Authenticated connection [id: 0xcbc657b4, /127.0.0.1:40782 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.041 [DEBUG - New I/O worker #2245] (Negotiator.java:764) Authenticated connection [id: 0x5cfaf1e1, /127.0.0.1:38786 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.037 [DEBUG - New I/O worker #2420] (Negotiator.java:764) Authenticated connection [id: 0x51f291a3, /127.0.0.1:40742 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.034 [DEBUG - New I/O worker #2393] (Negotiator.java:764) Authenticated connection [id: 0x6d216c28, /127.0.0.1:54134 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.034 [DEBUG - New I/O worker #2251] (Negotiator.java:764) Authenticated connection [id: 0x7fa30fdf, /127.0.0.1:38776 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.032 [DEBUG - New I/O worker #2242] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.115 [DEBUG - New I/O worker #2404] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.115 [DEBUG - New I/O worker #2535] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.105 [DEBUG - pool-3-thread-42] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.098 [DEBUG - New I/O worker #2317] (Negotiator.java:764) Authenticated connection [id: 0xdc0d12d7, /127.0.0.1:38804 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.098 [DEBUG - pool-3-thread-62] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.097 [DEBUG - New I/O worker #2262] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.146 [DEBUG - pool-3-thread-50] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.096 [DEBUG - New I/O worker #2276] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.148 [DEBUG - New I/O worker #2276] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=502), Traces: [0ms] refreshing cache from master, [90ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [96ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [103ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [420ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [451ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:42.148 [DEBUG - New I/O worker #2276] (Deferred.java:1330) callback=retry RPC@1702656950 returned Deferred@1631839169(state=PENDING, result=null, callback=(continuation of Deferred@608268685 after retry RPC@1702656950), errback=(continuation of Deferred@608268685 after retry RPC@1702656950)), so the following Deferred is getting paused: Deferred@608268685(state=PAUSED, result=Deferred@1631839169, callback=org.apache.kudu.client.AsyncKuduClient$6@13089bb4 -> wakeup thread pool-3-thread-5, errback=passthrough -> wakeup thread pool-3-thread-5) 00:27:42.096 [DEBUG - New I/O worker #2426] (Negotiator.java:764) Authenticated connection [id: 0x350fcefa, /127.0.0.1:54152 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.094 [DEBUG - New I/O worker #2286] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.149 [DEBUG - New I/O worker #2286] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=737), Traces: [0ms] refreshing cache from master, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [263ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [275ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [553ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [620ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [652ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.150 [DEBUG - New I/O worker #2286] (Deferred.java:1330) callback=retry RPC@1771315400 returned Deferred@1839966498(state=PENDING, result=null, callback=(continuation of Deferred@682175219 after retry RPC@1771315400), errback=(continuation of Deferred@682175219 after retry RPC@1771315400)), so the following Deferred is getting paused: Deferred@682175219(state=PAUSED, result=Deferred@1839966498, callback=org.apache.kudu.client.AsyncKuduClient$6@51913ce6 -> wakeup thread pool-3-thread-4, errback=passthrough -> wakeup thread pool-3-thread-4) 00:27:42.093 [DEBUG - New I/O worker #2294] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.091 [DEBUG - New I/O worker #2292] (Deferred.java:1330) callback=retry RPC@645885197 returned Deferred@574680346(state=PENDING, result=null, callback=(continuation of Deferred@1732405046 after retry RPC@645885197), errback=(continuation of Deferred@1732405046 after retry RPC@645885197)), so the following Deferred is getting paused: Deferred@1732405046(state=PAUSED, result=Deferred@574680346, callback=org.apache.kudu.client.AsyncKuduClient$6@ba53662 -> wakeup thread pool-3-thread-13, errback=passthrough -> wakeup thread pool-3-thread-13) 00:27:42.091 [DEBUG - New I/O worker #2314] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.088 [DEBUG - New I/O worker #2315] (NetUtil.java:106) Resolved IP of `127.22.19.252' to /127.22.19.252 in 5585156ns 00:27:42.152 [DEBUG - New I/O worker #2315] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.088 [DEBUG - New I/O worker #2416] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.087 [DEBUG - New I/O worker #2429] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.087 [DEBUG - New I/O worker #2250] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.087 [DEBUG - New I/O worker #2495] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.162 [DEBUG - New I/O worker #2315] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=502), Traces: [0ms] refreshing cache from master, [77ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [82ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [86ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [415ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.087 [DEBUG - New I/O worker #2344] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.087 [DEBUG - New I/O worker #2476] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.165 [DEBUG - New I/O worker #2260] (Negotiator.java:764) Authenticated connection [id: 0x552c2d00, /127.0.0.1:54172 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.087 [DEBUG - New I/O worker #2324] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.086 [DEBUG - New I/O worker #2427] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.085 [DEBUG - New I/O worker #2278] (Negotiator.java:764) Authenticated connection [id: 0xfb72ab2c, /127.0.0.1:38784 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.085 [DEBUG - New I/O worker #2255] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.083 [DEBUG - pool-3-thread-66] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.082 [DEBUG - New I/O worker #2451] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.082 [DEBUG - New I/O worker #2248] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.082 [DEBUG - New I/O worker #2496] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.082 [DEBUG - New I/O worker #2303] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.082 [DEBUG - New I/O worker #2267] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.082 [DEBUG - New I/O worker #2271] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.082 [DEBUG - New I/O worker #2289] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.082 [DEBUG - New I/O worker #2402] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.082 [DEBUG - New I/O worker #2259] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.179 [DEBUG - New I/O worker #2528] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.180 [DEBUG - New I/O worker #2333] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.082 [DEBUG - New I/O worker #2341] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.067 [DEBUG - New I/O worker #2423] (Negotiator.java:764) Authenticated connection [id: 0x4baf4d9c, /127.0.0.1:54160 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.312 [DEBUG - New I/O worker #2404] (Negotiator.java:764) Authenticated connection [id: 0x215b6e29, /127.0.0.1:40848 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.314 [DEBUG - New I/O worker #2322] (Negotiator.java:764) Authenticated connection [id: 0x84069ae3, /127.0.0.1:38820 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.307 [DEBUG - New I/O worker #2527] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.315 [DEBUG - New I/O worker #2253] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.305 [DEBUG - New I/O worker #2326] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.320 [DEBUG - New I/O worker #2322] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.320 [DEBUG - New I/O worker #2322] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=567), Traces: [0ms] refreshing cache from master, [130ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [130ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [131ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [567ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.321 [DEBUG - New I/O worker #2322] (Deferred.java:1330) callback=retry RPC@1922965562 returned Deferred@1990306481(state=PENDING, result=null, callback=(continuation of Deferred@866331137 after retry RPC@1922965562), errback=(continuation of Deferred@866331137 after retry RPC@1922965562)), so the following Deferred is getting paused: Deferred@866331137(state=PAUSED, result=Deferred@1990306481, callback=org.apache.kudu.client.AsyncKuduClient$6@1c6b27d -> wakeup thread pool-3-thread-53, errback=passthrough -> wakeup thread pool-3-thread-53) 00:27:42.323 [DEBUG - New I/O worker #2322] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.303 [DEBUG - New I/O worker #2316] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.301 [DEBUG - New I/O worker #2314] (Negotiator.java:764) Authenticated connection [id: 0x0a0a0088, /127.0.0.1:38858 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.301 [DEBUG - New I/O worker #2508] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.326 [DEBUG - New I/O worker #2322] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:42.326 [DEBUG - New I/O worker #2322] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:42.326 [DEBUG - New I/O worker #2322] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:42.326 [DEBUG - New I/O worker #2322] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:42.327 [DEBUG - New I/O worker #2322] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:42.298 [DEBUG - pool-3-thread-49] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.298 [DEBUG - New I/O worker #2256] (Negotiator.java:764) Authenticated connection [id: 0x03c102f3, /127.0.0.1:38826 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.298 [DEBUG - New I/O worker #2461] (Negotiator.java:764) Authenticated connection [id: 0xeefe3fcb, /127.0.0.1:40764 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.298 [DEBUG - New I/O worker #2375] (Negotiator.java:764) Authenticated connection [id: 0xcef00eee, /127.0.0.1:54138 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.296 [DEBUG - New I/O worker #2348] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.296 [DEBUG - pool-3-thread-1] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.294 [DEBUG - pool-3-thread-20] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.293 [DEBUG - pool-3-thread-40] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.293 [DEBUG - New I/O worker #2495] (Negotiator.java:764) Authenticated connection [id: 0x324b253a, /127.0.0.1:54190 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.293 [DEBUG - pool-3-thread-65] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.291 [DEBUG - New I/O worker #2311] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.290 [DEBUG - New I/O worker #2286] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.290 [DEBUG - New I/O worker #2250] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=536), Traces: [0ms] refreshing cache from master, [8ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [9ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [12ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [176ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [389ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [468ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, deferred=null) after lookup 00:27:42.334 [DEBUG - New I/O worker #2250] (Deferred.java:1330) callback=retry RPC@1304874553 returned Deferred@1241076569(state=PENDING, result=null, callback=(continuation of Deferred@217826306 after retry RPC@1304874553), errback=(continuation of Deferred@217826306 after retry RPC@1304874553)), so the following Deferred is getting paused: Deferred@217826306(state=PAUSED, result=Deferred@1241076569, callback=org.apache.kudu.client.AsyncKuduClient$6@467165c4 -> wakeup thread pool-3-thread-37, errback=passthrough -> wakeup thread pool-3-thread-37) 00:27:42.290 [DEBUG - New I/O worker #2265] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.290 [DEBUG - New I/O worker #2243] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.283 [DEBUG - New I/O worker #2309] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.281 [DEBUG - New I/O worker #2278] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.336 [DEBUG - New I/O worker #2278] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=690), Traces: [0ms] refreshing cache from master, [78ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [82ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [83ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [474ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [635ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.337 [DEBUG - New I/O worker #2278] (Deferred.java:1330) callback=retry RPC@148986432 returned Deferred@215932804(state=PENDING, result=null, callback=(continuation of Deferred@1239175291 after retry RPC@148986432), errback=(continuation of Deferred@1239175291 after retry RPC@148986432)), so the following Deferred is getting paused: Deferred@1239175291(state=PAUSED, result=Deferred@215932804, callback=org.apache.kudu.client.AsyncKuduClient$6@7aeecca0 -> wakeup thread pool-3-thread-22, errback=passthrough -> wakeup thread pool-3-thread-22) 00:27:42.276 [DEBUG - New I/O worker #2319] (Negotiator.java:764) Authenticated connection [id: 0x51c3cffc, /127.0.0.1:54140 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.275 [DEBUG - New I/O worker #2335] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.275 [DEBUG - New I/O worker #2313] (Negotiator.java:764) Authenticated connection [id: 0xbe94cef6, /127.0.0.1:38814 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.275 [DEBUG - New I/O worker #2301] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.275 [DEBUG - New I/O worker #2509] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.273 [DEBUG - New I/O worker #2486] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.272 [DEBUG - New I/O worker #2339] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.201 [DEBUG - pool-3-thread-17] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.184 [DEBUG - New I/O worker #2529] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.183 [DEBUG - New I/O worker #2274] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.163 [DEBUG - New I/O worker #2276] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.163 [DEBUG - New I/O worker #2292] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.163 [DEBUG - New I/O worker #2315] (Deferred.java:1330) callback=retry RPC@176820625 returned Deferred@246776856(state=PENDING, result=null, callback=(continuation of Deferred@1270291370 after retry RPC@176820625), errback=(continuation of Deferred@1270291370 after retry RPC@176820625)), so the following Deferred is getting paused: Deferred@1270291370(state=PAUSED, result=Deferred@246776856, callback=org.apache.kudu.client.AsyncKuduClient$6@20ea9b1b -> wakeup thread pool-3-thread-3, errback=passthrough -> wakeup thread pool-3-thread-3) 00:27:42.347 [DEBUG - New I/O worker #2315] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.162 [DEBUG - New I/O worker #2317] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.348 [DEBUG - New I/O worker #2317] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=739), Traces: [0ms] refreshing cache from master, [132ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [132ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [139ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [488ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [545ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, [555ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, deferred=null) after lookup 00:27:42.348 [DEBUG - New I/O worker #2317] (Deferred.java:1330) callback=retry RPC@483687268 returned Deferred@418114598(state=PENDING, result=null, callback=(continuation of Deferred@1575579999 after retry RPC@483687268), errback=(continuation of Deferred@1575579999 after retry RPC@483687268)), so the following Deferred is getting paused: Deferred@1575579999(state=PAUSED, result=Deferred@418114598, callback=org.apache.kudu.client.AsyncKuduClient$6@534e7483 -> wakeup thread pool-3-thread-46, errback=passthrough -> wakeup thread pool-3-thread-46) 00:27:42.162 [DEBUG - New I/O worker #2334] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.162 [DEBUG - New I/O worker #2251] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.349 [DEBUG - New I/O worker #2251] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=961), Traces: [0ms] refreshing cache from master, [297ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [305ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [644ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [679ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [751ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.350 [DEBUG - New I/O worker #2251] (Deferred.java:1330) callback=retry RPC@2244603 returned Deferred@69053223(state=PENDING, result=null, callback=(continuation of Deferred@1092564416 after retry RPC@2244603), errback=(continuation of Deferred@1092564416 after retry RPC@2244603)), so the following Deferred is getting paused: Deferred@1092564416(state=PAUSED, result=Deferred@69053223, callback=org.apache.kudu.client.AsyncKuduClient$6@61d8b2cc -> wakeup thread pool-3-thread-29, errback=passthrough -> wakeup thread pool-3-thread-29) 00:27:42.156 [DEBUG - New I/O worker #2390] (Negotiator.java:764) Authenticated connection [id: 0x7abd2f02, /127.0.0.1:40770 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.143 [DEBUG - New I/O worker #2386] (Negotiator.java:764) Authenticated connection [id: 0xbbdc4855, /127.0.0.1:40784 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.133 [DEBUG - pool-3-thread-39] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.131 [DEBUG - New I/O worker #2245] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.352 [DEBUG - New I/O worker #2245] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=705), Traces: [0ms] refreshing cache from master, [78ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [86ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [86ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [459ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.352 [DEBUG - New I/O worker #2245] (Deferred.java:1330) callback=retry RPC@1897566378 returned Deferred@1965361745(state=PENDING, result=null, callback=(continuation of Deferred@807901841 after retry RPC@1897566378), errback=(continuation of Deferred@807901841 after retry RPC@1897566378)), so the following Deferred is getting paused: Deferred@807901841(state=PAUSED, result=Deferred@1965361745, callback=org.apache.kudu.client.AsyncKuduClient$6@246731dd -> wakeup thread pool-3-thread-33, errback=passthrough -> wakeup thread pool-3-thread-33) 00:27:42.123 [DEBUG - pool-3-thread-43] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.122 [DEBUG - New I/O worker #2377] (Negotiator.java:764) Authenticated connection [id: 0xa031c81d, /127.0.0.1:40772 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.359 [DEBUG - New I/O worker #2251] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.366 [DEBUG - New I/O worker #2251] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:42.366 [DEBUG - New I/O worker #2251] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:42.366 [DEBUG - New I/O worker #2251] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:42.366 [DEBUG - New I/O worker #2251] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:42.366 [DEBUG - New I/O worker #2251] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:42.367 [DEBUG - pool-3-thread-29] (Deferred.java:1330) callback=disconnect callback@2134679649 returned Deferred@1741582064(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1576143180 after disconnect callback@2134679649), errback=passthrough -> (continuation of Deferred@1576143180 after disconnect callback@2134679649)), so the following Deferred is getting paused: Deferred@1576143180(state=PAUSED, result=Deferred@1741582064, callback=, errback=) 00:27:42.368 [DEBUG - New I/O worker #2269] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:42.368 [DEBUG - New I/O worker #2327] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:42.369 [DEBUG - New I/O worker #2251] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:42.373 [DEBUG - New I/O worker #2276] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:42.373 [DEBUG - New I/O worker #2276] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:42.373 [DEBUG - New I/O worker #2276] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:42.373 [DEBUG - New I/O worker #2276] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:42.374 [DEBUG - New I/O worker #2276] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:42.374 [DEBUG - pool-3-thread-5] (Deferred.java:1330) callback=disconnect callback@2089651217 returned Deferred@501155356(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1731919365 after disconnect callback@2089651217), errback=passthrough -> (continuation of Deferred@1731919365 after disconnect callback@2089651217)), so the following Deferred is getting paused: Deferred@1731919365(state=PAUSED, result=Deferred@501155356, callback=, errback=) 00:27:42.374 [DEBUG - New I/O worker #2277] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:42.375 [DEBUG - New I/O worker #2386] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:42.375 [INFO - New I/O worker #2386] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:42.375 [DEBUG - New I/O worker #2276] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:42.400 [DEBUG - New I/O worker #2527] (Negotiator.java:764) Authenticated connection [id: 0x900b6bff, /127.0.0.1:40894 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.403 [DEBUG - New I/O worker #2341] (Negotiator.java:764) Authenticated connection [id: 0x19c67359, /127.0.0.1:40840 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.407 [DEBUG - New I/O worker #2264] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.410 [DEBUG - New I/O worker #2331] (Negotiator.java:764) Authenticated connection [id: 0x27f98f83, /127.0.0.1:40802 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.413 [DEBUG - New I/O worker #2245] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.413 [DEBUG - New I/O worker #2301] (Negotiator.java:764) Authenticated connection [id: 0xc57c3af2, /127.0.0.1:38900 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.414 [DEBUG - New I/O worker #2274] (Negotiator.java:764) Authenticated connection [id: 0x208f4d08, /127.0.0.1:54238 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.417 [DEBUG - New I/O worker #2316] (Negotiator.java:764) Authenticated connection [id: 0x54b2eb90, /127.0.0.1:38912 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:42.417604 26818 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:42.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.887007 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40796 00:27:42.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.887337 (+ 330us) server_negotiation.cc:184] Beginning negotiation 00:27:42.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.887390 (+ 53us) server_negotiation.cc:373] Waiting for connection header 00:27:42.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.887435 (+ 45us) server_negotiation.cc:381] Connection header received 00:27:42.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.887544 (+ 109us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:42.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.887575 (+ 31us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:42.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.887702 (+ 127us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:42.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.887868 (+ 166us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:42.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.037806 (+149938us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:42.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.038944 (+ 1138us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:42.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.201780 (+162836us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:42.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.202989 (+ 1209us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:42.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.203180 (+ 191us) server_negotiation.cc:589] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 00:27:42.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.416507 (+213327us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40796: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:40796): Connection reset by peer (error 104) 00:27:42.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":112} 00:27:42.422 [DEBUG - New I/O worker #2325] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state NEGOTIATING due to: connection disconnected 00:27:42.422 [INFO - New I/O worker #2325] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.254:44849: connection disconnected 00:27:42.423 [DEBUG - New I/O worker #2322] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:42.424942 26337 negotiation.cc:313] Failed RPC negotiation. Trace: 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.886652 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:54166 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.887000 (+ 348us) server_negotiation.cc:184] Beginning negotiation 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.887016 (+ 16us) server_negotiation.cc:373] Waiting for connection header 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.901247 (+ 14231us) server_negotiation.cc:381] Connection header received 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.901366 (+ 119us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.901404 (+ 38us) server_negotiation.cc:420] Received NEGOTIATE request from client 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.901587 (+ 183us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:41.901742 (+ 155us) server_negotiation.cc:205] Negotiated authn=SASL 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.079121 (+177379us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.080350 (+ 1229us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.322593 (+242243us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.389048 (+ 66455us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.389240 (+ 192us) server_negotiation.cc:589] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.397576 (+ 8336us) server_negotiation.cc:337] Received SASL_INITIATE NegotiatePB request 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.397607 (+ 31us) server_negotiation.cc:746] Received SASL_INITIATE request from client 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.397631 (+ 24us) server_negotiation.cc:757] Client requested to use mechanism: PLAIN 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.397693 (+ 62us) server_negotiation.cc:786] Calling sasl_server_start() 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.397769 (+ 76us) server_negotiation.cc:945] Received PLAIN auth, user=slave 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.397854 (+ 85us) server_negotiation.cc:349] Sending SASL_SUCCESS NegotiatePB response 00:27:42.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.398080 (+ 226us) server_negotiation.cc:890] Waiting for connection context 00:27:42.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0220 00:27:42.423413 (+ 25333us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54166: BlockingRecv error: failed to read from TLS socket (remote: 0.0.0.0:0): Cannot send after transport endpoint shutdown (error 108) 00:27:42.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":33024,"server-negotiator.queue_time_us":96} 00:27:42.433 [DEBUG - New I/O worker #2309] (Negotiator.java:764) Authenticated connection [id: 0x52f1f9b3, /127.0.0.1:38898 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.449 [DEBUG - New I/O worker #2317] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.456 [DEBUG - New I/O worker #2317] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:42.456 [DEBUG - New I/O worker #2317] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:42.456 [DEBUG - New I/O worker #2317] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:42.457 [DEBUG - New I/O worker #2317] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:42.457 [DEBUG - New I/O worker #2317] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:42.458 [DEBUG - pool-3-thread-46] (Deferred.java:1330) callback=disconnect callback@1473578139 returned Deferred@503346807(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@637705 after disconnect callback@1473578139), errback=passthrough -> (continuation of Deferred@637705 after disconnect callback@1473578139)), so the following Deferred is getting paused: Deferred@637705(state=PAUSED, result=Deferred@503346807, callback=, errback=) 00:27:42.458 [DEBUG - New I/O worker #2426] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:42.458 [DEBUG - New I/O worker #2430] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:42.456 [DEBUG - New I/O worker #2301] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.460 [DEBUG - New I/O worker #2301] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=296), Traces: [0ms] refreshing cache from master, [105ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [106ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [106ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [294ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.460 [DEBUG - New I/O worker #2301] (Deferred.java:1330) callback=retry RPC@202243454 returned Deferred@137521191(state=PENDING, result=null, callback=(continuation of Deferred@1295054661 after retry RPC@202243454), errback=(continuation of Deferred@1295054661 after retry RPC@202243454)), so the following Deferred is getting paused: Deferred@1295054661(state=PAUSED, result=Deferred@137521191, callback=org.apache.kudu.client.AsyncKuduClient$6@1fb2850f -> wakeup thread pool-3-thread-50, errback=passthrough -> wakeup thread pool-3-thread-50) 00:27:42.461 [DEBUG - New I/O worker #2317] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:42.465 [DEBUG - New I/O worker #2324] (Negotiator.java:764) Authenticated connection [id: 0xe212e3b3, /127.0.0.1:54194 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.467 [DEBUG - New I/O worker #2255] (Negotiator.java:764) Authenticated connection [id: 0x11bf0d3c, /127.0.0.1:54196 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.468 [DEBUG - pool-3-thread-28] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.470 [DEBUG - pool-3-thread-77] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.471 [INFO - New I/O worker #2503] (Connection.java:435) [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:42.471 [DEBUG - New I/O worker #2503] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state NEGOTIATING due to: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:42.472 [INFO - New I/O worker #2503] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: [peer master-127.22.19.253:37879(127.22.19.253:37879)] disconnected from peer 00:27:42.472 [DEBUG - New I/O worker #2265] (Negotiator.java:764) Authenticated connection [id: 0xcd5b4304, /127.0.0.1:38880 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.472 [DEBUG - pool-3-thread-55] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.473 [DEBUG - pool-3-thread-16] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.474 [DEBUG - New I/O worker #2289] (Negotiator.java:764) Authenticated connection [id: 0x5a5b691e, /127.0.0.1:38836 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.474 [DEBUG - pool-3-thread-35] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.475 [DEBUG - New I/O worker #2256] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.475 [DEBUG - pool-3-thread-11] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.475 [DEBUG - New I/O worker #2256] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=681), Traces: [0ms] refreshing cache from master, [112ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [114ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [114ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [374ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [681ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.476 [DEBUG - New I/O worker #2256] (Deferred.java:1330) callback=retry RPC@2118186104 returned Deferred@2055176040(state=PENDING, result=null, callback=(continuation of Deferred@1065221699 after retry RPC@2118186104), errback=(continuation of Deferred@1065221699 after retry RPC@2118186104)), so the following Deferred is getting paused: Deferred@1065221699(state=PAUSED, result=Deferred@2055176040, callback=org.apache.kudu.client.AsyncKuduClient$6@77920a60 -> wakeup thread pool-3-thread-25, errback=passthrough -> wakeup thread pool-3-thread-25) 00:27:42.475 [DEBUG - pool-3-thread-8] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.476 [DEBUG - New I/O worker #2253] (Negotiator.java:764) Authenticated connection [id: 0xfda9d05d, /127.0.0.1:38918 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.477 [DEBUG - pool-3-thread-56] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.477 [DEBUG - New I/O worker #2314] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.477 [DEBUG - New I/O worker #2314] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=439), Traces: [0ms] refreshing cache from master, [15ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [27ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [27ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [439ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.477 [DEBUG - pool-3-thread-72] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.478 [DEBUG - pool-3-thread-7] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.479 [DEBUG - Time-limited test] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.476 [DEBUG - pool-3-thread-45] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.483 [DEBUG - pool-3-thread-51] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.482 [DEBUG - pool-3-thread-54] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.482 [DEBUG - pool-3-thread-61] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.481 [DEBUG - pool-3-thread-79] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.481 [DEBUG - pool-3-thread-31] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.480 [DEBUG - pool-3-thread-71] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.480 [DEBUG - pool-3-thread-68] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.479 [DEBUG - pool-3-thread-58] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:42.477 [DEBUG - New I/O worker #2314] (Deferred.java:1330) callback=retry RPC@1572109643 returned Deferred@1502343717(state=PENDING, result=null, callback=(continuation of Deferred@478771056 after retry RPC@1572109643), errback=(continuation of Deferred@478771056 after retry RPC@1572109643)), so the following Deferred is getting paused: Deferred@478771056(state=PAUSED, result=Deferred@1502343717, callback=org.apache.kudu.client.AsyncKuduClient$6@537ffc85 -> wakeup thread pool-3-thread-52, errback=passthrough -> wakeup thread pool-3-thread-52) 00:27:42.477 [DEBUG - New I/O worker #2313] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.486 [DEBUG - New I/O worker #2313] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=877), Traces: [0ms] refreshing cache from master, [135ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [144ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [145ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [429ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [719ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [869ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.486 [DEBUG - New I/O worker #2313] (Deferred.java:1330) callback=retry RPC@2139317079 returned Deferred@2075977855(state=PENDING, result=null, callback=(continuation of Deferred@1052666220 after retry RPC@2139317079), errback=(continuation of Deferred@1052666220 after retry RPC@2139317079)), so the following Deferred is getting paused: Deferred@1052666220(state=PAUSED, result=Deferred@2075977855, callback=org.apache.kudu.client.AsyncKuduClient$6@6b37c94f -> wakeup thread pool-3-thread-69, errback=passthrough -> wakeup thread pool-3-thread-69) 00:27:42.489 [DEBUG - New I/O worker #2535] (Negotiator.java:764) Authenticated connection [id: 0xca4e1e3c, /127.0.0.1:40850 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.494 [DEBUG - New I/O worker #2267] (Negotiator.java:764) Authenticated connection [id: 0x750678bd, /127.0.0.1:38840 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.497 [DEBUG - New I/O worker #2496] (Negotiator.java:764) Authenticated connection [id: 0xbb62d118, /127.0.0.1:40828 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.498 [DEBUG - New I/O worker #2402] (Negotiator.java:764) Authenticated connection [id: 0xf6094a9b, /127.0.0.1:54198 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.498 [DEBUG - New I/O worker #2262] (Negotiator.java:764) Authenticated connection [id: 0xbebf32f4, /127.0.0.1:40842 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.499 [DEBUG - New I/O worker #2245] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:42.499 [DEBUG - New I/O worker #2245] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:42.499 [DEBUG - New I/O worker #2245] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:42.500 [DEBUG - New I/O worker #2245] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:42.500 [DEBUG - New I/O worker #2245] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:42.500 [DEBUG - pool-3-thread-33] (Deferred.java:1330) callback=disconnect callback@1581022870 returned Deferred@897272586(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1177961015 after disconnect callback@1581022870), errback=passthrough -> (continuation of Deferred@1177961015 after disconnect callback@1581022870)), so the following Deferred is getting paused: Deferred@1177961015(state=PAUSED, result=Deferred@897272586, callback=, errback=) 00:27:42.501 [DEBUG - New I/O worker #2390] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:42.501 [INFO - New I/O worker #2390] (ConnectToCluster.java:361) Unable to connect to master 127.22.19.253:37879: connection disconnected 00:27:42.501 [DEBUG - New I/O worker #2319] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:42.502 [DEBUG - New I/O worker #2245] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:42.503 [DEBUG - New I/O worker #2528] (Negotiator.java:764) Authenticated connection [id: 0x1af6449b, /127.0.0.1:40858 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.504 [DEBUG - New I/O worker #2294] (Negotiator.java:764) Authenticated connection [id: 0x407a3d3c, /127.0.0.1:38860 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.512 [DEBUG - New I/O worker #2242] (Negotiator.java:764) Authenticated connection [id: 0x7fa7ce2e, /127.0.0.1:38832 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.515 [DEBUG - New I/O worker #2248] (Negotiator.java:764) Authenticated connection [id: 0xe0b8e11c, /127.0.0.1:38842 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:42.517391 22692 connection.cc:554] Connection torn down before Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:40784 (request call id 0) could send its response 00:27:42.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:42.518811 22692 connection.cc:554] Connection torn down before Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:40770 (request call id 0) could send its response 00:27:42.519 [DEBUG - New I/O worker #2476] (Negotiator.java:764) Authenticated connection [id: 0xb84a5ae0, /127.0.0.1:40836 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.519 [DEBUG - New I/O worker #2271] (Negotiator.java:764) Authenticated connection [id: 0xa2e0e6bf, /127.0.0.1:38844 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.520 [DEBUG - New I/O worker #2243] (Negotiator.java:764) Authenticated connection [id: 0x50524f5c, /127.0.0.1:38892 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.520 [DEBUG - New I/O worker #2311] (Negotiator.java:764) Authenticated connection [id: 0xbdd56fe9, /127.0.0.1:38882 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.525 [DEBUG - New I/O worker #2529] (Negotiator.java:764) Authenticated connection [id: 0x11029e2e, /127.0.0.1:40862 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.526 [DEBUG - New I/O worker #2486] (Negotiator.java:764) Authenticated connection [id: 0x100f3454, /127.0.0.1:40880 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.530 [DEBUG - New I/O worker #2314] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.531 [DEBUG - New I/O worker #2509] (Negotiator.java:764) Authenticated connection [id: 0x94a5d4f0, /127.0.0.1:40882 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.533 [DEBUG - New I/O worker #2429] (Negotiator.java:764) Authenticated connection [id: 0x2348c836, /127.0.0.1:40846 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.534 [DEBUG - New I/O worker #2508] (Negotiator.java:764) Authenticated connection [id: 0xe20e5050, /127.0.0.1:40888 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.537 [DEBUG - New I/O worker #2333] (Negotiator.java:764) Authenticated connection [id: 0x9f8be612, /127.0.0.1:54232 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.543 [DEBUG - New I/O worker #2256] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.544 [DEBUG - New I/O worker #2259] (Negotiator.java:764) Authenticated connection [id: 0xac775fe5, /127.0.0.1:54210 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.545 [DEBUG - New I/O worker #2427] (Negotiator.java:764) Authenticated connection [id: 0x905ea0fb, /127.0.0.1:54216 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.545 [DEBUG - New I/O worker #2309] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.546 [DEBUG - New I/O worker #2309] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=375), Traces: [0ms] refreshing cache from master, [97ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [98ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [98ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [359ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [375ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.546 [DEBUG - New I/O worker #2309] (Deferred.java:1330) callback=retry RPC@1059522003 returned Deferred@991467247(state=PENDING, result=null, callback=(continuation of Deferred@2115637224 after retry RPC@1059522003), errback=(continuation of Deferred@2115637224 after retry RPC@1059522003)), so the following Deferred is getting paused: Deferred@2115637224(state=PAUSED, result=Deferred@991467247, callback=org.apache.kudu.client.AsyncKuduClient$6@482019b1 -> wakeup thread pool-3-thread-66, errback=passthrough -> wakeup thread pool-3-thread-66) 00:27:42.548 [DEBUG - New I/O worker #2315] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:42.548 [DEBUG - New I/O worker #2315] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:42.548 [DEBUG - New I/O worker #2315] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:42.549 [DEBUG - New I/O worker #2315] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:42.549 [DEBUG - New I/O worker #2315] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:42.549 [DEBUG - New I/O worker #2303] (Negotiator.java:764) Authenticated connection [id: 0x340b1b71, /127.0.0.1:38834 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.550 [DEBUG - pool-3-thread-3] (Deferred.java:1330) callback=disconnect callback@1931402693 returned Deferred@1969680722(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2072431916 after disconnect callback@1931402693), errback=passthrough -> (continuation of Deferred@2072431916 after disconnect callback@1931402693)), so the following Deferred is getting paused: Deferred@2072431916(state=PAUSED, result=Deferred@1969680722, callback=, errback=) 00:27:42.550 [DEBUG - New I/O worker #2377] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:42.550 [DEBUG - New I/O worker #2315] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:42.548 [DEBUG - New I/O worker #2344] (Negotiator.java:764) Authenticated connection [id: 0x6217e542, /127.0.0.1:54206 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.550 [DEBUG - New I/O worker #2375] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:42.549 [DEBUG - New I/O worker #2416] (Negotiator.java:764) Authenticated connection [id: 0x709b3a0e, /127.0.0.1:54182 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.551 [DEBUG - New I/O worker #2334] (Negotiator.java:764) Authenticated connection [id: 0x667f1af1, /127.0.0.1:40868 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.552 [DEBUG - New I/O worker #2313] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.554 [DEBUG - New I/O worker #2301] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.555 [DEBUG - New I/O worker #2451] (Negotiator.java:764) Authenticated connection [id: 0x9ee75d32, /127.0.0.1:40820 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.559 [DEBUG - New I/O worker #2335] (Negotiator.java:764) Authenticated connection [id: 0x481daf91, /127.0.0.1:54228 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.560 [DEBUG - New I/O worker #2294] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.560 [DEBUG - New I/O worker #2294] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=523), Traces: [0ms] refreshing cache from master, [20ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [24ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [24ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [487ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [521ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [523ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.561 [DEBUG - New I/O worker #2294] (Deferred.java:1330) callback=retry RPC@511409650 returned Deferred@440728293(state=PENDING, result=null, callback=(continuation of Deferred@1598453705 after retry RPC@511409650), errback=(continuation of Deferred@1598453705 after retry RPC@511409650)), so the following Deferred is getting paused: Deferred@1598453705(state=PAUSED, result=Deferred@440728293, callback=org.apache.kudu.client.AsyncKuduClient$6@2b02758d -> wakeup thread pool-3-thread-27, errback=passthrough -> wakeup thread pool-3-thread-27) 00:27:42.562 [DEBUG - New I/O worker #2339] (Negotiator.java:764) Authenticated connection [id: 0xa857a775, /127.0.0.1:54246 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.564 [DEBUG - New I/O worker #2326] (Negotiator.java:764) Authenticated connection [id: 0xaeb28ded, /127.0.0.1:54258 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.564 [DEBUG - New I/O worker #2242] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.564 [DEBUG - New I/O worker #2242] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=732), Traces: [0ms] refreshing cache from master, [151ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [230ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [231ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [578ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [716ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [731ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.564 [DEBUG - New I/O worker #2242] (Deferred.java:1330) callback=retry RPC@1319966108 returned Deferred@1251121297(state=PENDING, result=null, callback=(continuation of Deferred@261103527 after retry RPC@1319966108), errback=(continuation of Deferred@261103527 after retry RPC@1319966108)), so the following Deferred is getting paused: Deferred@261103527(state=PAUSED, result=Deferred@1251121297, callback=org.apache.kudu.client.AsyncKuduClient$6@18f2104f -> wakeup thread pool-3-thread-36, errback=passthrough -> wakeup thread pool-3-thread-36) 00:27:42.566 [DEBUG - New I/O worker #2264] (Negotiator.java:764) Authenticated connection [id: 0x4d37baef, /127.0.0.1:54264 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.567 [DEBUG - New I/O worker #2348] (Negotiator.java:764) Authenticated connection [id: 0x2cdb0252, /127.0.0.1:54248 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.568 [DEBUG - New I/O worker #2253] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.568 [DEBUG - New I/O worker #2267] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.569 [DEBUG - New I/O worker #2267] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=531), Traces: [0ms] refreshing cache from master, [6ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [12ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [30ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [455ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [498ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [530ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.569 [DEBUG - New I/O worker #2267] (Deferred.java:1330) callback=retry RPC@12136518 returned Deferred@75936905(state=PENDING, result=null, callback=(continuation of Deferred@1099184765 after retry RPC@12136518), errback=(continuation of Deferred@1099184765 after retry RPC@12136518)), so the following Deferred is getting paused: Deferred@1099184765(state=PAUSED, result=Deferred@75936905, callback=org.apache.kudu.client.AsyncKuduClient$6@397da4a3 -> wakeup thread pool-3-thread-9, errback=passthrough -> wakeup thread pool-3-thread-9) 00:27:42.570 [DEBUG - New I/O worker #2253] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=386), Traces: [0ms] refreshing cache from master, [93ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [94ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [94ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [360ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [385ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.570 [DEBUG - New I/O worker #2253] (Deferred.java:1330) callback=retry RPC@971803862 returned Deferred@1037246259(state=PENDING, result=null, callback=(continuation of Deferred@2026997485 after retry RPC@971803862), errback=(continuation of Deferred@2026997485 after retry RPC@971803862)), so the following Deferred is getting paused: Deferred@2026997485(state=PAUSED, result=Deferred@1037246259, callback=org.apache.kudu.client.AsyncKuduClient$6@14347cd8 -> wakeup thread pool-3-thread-62, errback=passthrough -> wakeup thread pool-3-thread-62) 00:27:42.572 [DEBUG - New I/O worker #2315] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:42.572 [DEBUG - New I/O worker #2248] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.572 [DEBUG - New I/O worker #2248] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=540), Traces: [0ms] refreshing cache from master, [13ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [19ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [31ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [493ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [505ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [540ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.572 [DEBUG - New I/O worker #2248] (Deferred.java:1330) callback=retry RPC@1210546552 returned Deferred@1276703908(state=PENDING, result=null, callback=(continuation of Deferred@152728387 after retry RPC@1210546552), errback=(continuation of Deferred@152728387 after retry RPC@1210546552)), so the following Deferred is getting paused: Deferred@152728387(state=PAUSED, result=Deferred@1276703908, callback=org.apache.kudu.client.AsyncKuduClient$6@4bc6538b -> wakeup thread pool-3-thread-6, errback=passthrough -> wakeup thread pool-3-thread-6) 00:27:42.573 [DEBUG - New I/O worker #2314] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:42.573 [DEBUG - New I/O worker #2314] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:42.574 [DEBUG - New I/O worker #2314] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:42.574 [DEBUG - New I/O worker #2314] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:42.574 [DEBUG - New I/O worker #2314] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:42.574 [DEBUG - New I/O worker #2256] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:42.574 [DEBUG - New I/O worker #2256] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:42.575 [DEBUG - New I/O worker #2256] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:42.575 [DEBUG - New I/O worker #2256] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:42.576 [DEBUG - New I/O worker #2256] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:42.576 [DEBUG - pool-3-thread-25] (Deferred.java:1330) callback=disconnect callback@423868380 returned Deferred@1061736584(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1202058524 after disconnect callback@423868380), errback=passthrough -> (continuation of Deferred@1202058524 after disconnect callback@423868380)), so the following Deferred is getting paused: Deferred@1202058524(state=PAUSED, result=Deferred@1061736584, callback=, errback=) 00:27:42.576 [DEBUG - New I/O worker #2260] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:42.577 [DEBUG - New I/O worker #2245] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:42.575 [DEBUG - New I/O worker #2345] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.577 [DEBUG - New I/O worker #2331] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:42.577 [DEBUG - New I/O worker #2488] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.578 [DEBUG - New I/O worker #2301] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:42.579 [DEBUG - New I/O worker #2301] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:42.579 [DEBUG - New I/O worker #2301] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:42.579 [DEBUG - New I/O worker #2301] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:42.576 [DEBUG - New I/O worker #2243] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.575 [DEBUG - pool-3-thread-52] (Deferred.java:1330) callback=disconnect callback@406605143 returned Deferred@1440466447(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@416457463 after disconnect callback@406605143), errback=passthrough -> (continuation of Deferred@416457463 after disconnect callback@406605143)), so the following Deferred is getting paused: Deferred@416457463(state=PAUSED, result=Deferred@1440466447, callback=, errback=) 00:27:42.579 [DEBUG - New I/O worker #2301] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:42.580 [DEBUG - pool-3-thread-50] (Deferred.java:1330) callback=disconnect callback@1901960776 returned Deferred@1280331807(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@976715027 after disconnect callback@1901960776), errback=passthrough -> (continuation of Deferred@976715027 after disconnect callback@1901960776)), so the following Deferred is getting paused: Deferred@976715027(state=PAUSED, result=Deferred@1280331807, callback=, errback=) 00:27:42.581 [DEBUG - New I/O worker #2301] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:42.579 [DEBUG - New I/O worker #2256] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:42.578 [DEBUG - New I/O worker #2314] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:42.578 [DEBUG - New I/O worker #2311] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.582 [DEBUG - New I/O worker #2311] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=493), Traces: [0ms] refreshing cache from master, [11ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [11ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [18ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [441ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [461ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [489ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.582 [DEBUG - New I/O worker #2311] (Deferred.java:1330) callback=retry RPC@941165672 returned Deferred@1009160115(state=PENDING, result=null, callback=(continuation of Deferred@2032404051 after retry RPC@941165672), errback=(continuation of Deferred@2032404051 after retry RPC@941165672)), so the following Deferred is getting paused: Deferred@2032404051(state=PAUSED, result=Deferred@1009160115, callback=org.apache.kudu.client.AsyncKuduClient$6@389d0d48 -> wakeup thread pool-3-thread-21, errback=passthrough -> wakeup thread pool-3-thread-21) 00:27:42.584 [DEBUG - New I/O worker #2483] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.584 [DEBUG - New I/O worker #2242] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.587 [DEBUG - New I/O worker #2243] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=508), Traces: [0ms] refreshing cache from master, [31ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [34ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [44ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [338ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [493ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [498ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.581 [DEBUG - New I/O worker #2309] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.587 [DEBUG - New I/O worker #2243] (Deferred.java:1330) callback=retry RPC@1655050469 returned Deferred@1721337961(state=PENDING, result=null, callback=(continuation of Deferred@597367518 after retry RPC@1655050469), errback=(continuation of Deferred@597367518 after retry RPC@1655050469)), so the following Deferred is getting paused: Deferred@597367518(state=PAUSED, result=Deferred@1721337961, callback=org.apache.kudu.client.AsyncKuduClient$6@40815c9f -> wakeup thread pool-3-thread-38, errback=passthrough -> wakeup thread pool-3-thread-38) 00:27:42.589 [DEBUG - New I/O worker #2429] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:42.590 [DEBUG - New I/O worker #2509] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:42.590 [DEBUG - New I/O worker #2427] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:42.591 [DEBUG - New I/O worker #2339] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:42.598 [DEBUG - New I/O worker #2350] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.598 [DEBUG - New I/O worker #2488] (Negotiator.java:764) Authenticated connection [id: 0x5f99077b, /127.0.0.1:40900 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.604 [DEBUG - New I/O worker #2345] (Negotiator.java:764) Authenticated connection [id: 0xfbe74682, /127.0.0.1:54270 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.604 [DEBUG - New I/O worker #2347] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.606 [DEBUG - New I/O worker #2484] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.612 [DEBUG - New I/O worker #2493] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.615 [DEBUG - New I/O worker #2336] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.616 [DEBUG - New I/O worker #2492] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.617 [DEBUG - New I/O worker #2261] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.621 [DEBUG - New I/O worker #2352] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.627 [DEBUG - New I/O worker #2483] (Negotiator.java:764) Authenticated connection [id: 0x2259be8c, /127.0.0.1:40906 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.627 [DEBUG - New I/O worker #2346] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.627 [DEBUG - New I/O worker #2586] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.629 [DEBUG - New I/O worker #2475] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.640 [DEBUG - New I/O worker #2300] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.644 [DEBUG - New I/O worker #2586] (Negotiator.java:764) Authenticated connection [id: 0x726be995, /127.0.0.1:40936 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.645 [DEBUG - New I/O worker #2254] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.654 [DEBUG - New I/O worker #2342] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.657 [DEBUG - New I/O worker #2487] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.660 [DEBUG - New I/O worker #2475] (Negotiator.java:764) Authenticated connection [id: 0x5312c194, /127.0.0.1:40930 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.662 [DEBUG - New I/O worker #2490] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.663 [DEBUG - New I/O worker #2346] (Negotiator.java:764) Authenticated connection [id: 0xb79833a3, /127.0.0.1:54306 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.671 [DEBUG - New I/O worker #2647] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.673 [DEBUG - New I/O worker #2492] (Negotiator.java:764) Authenticated connection [id: 0x98e859b4, /127.0.0.1:54288 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.673 [DEBUG - New I/O worker #2352] (Negotiator.java:764) Authenticated connection [id: 0x38ca4f90, /127.0.0.1:54300 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.682 [DEBUG - New I/O worker #2336] (Negotiator.java:764) Authenticated connection [id: 0x862f0833, /127.0.0.1:40924 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.687 [DEBUG - New I/O worker #2300] (Negotiator.java:764) Authenticated connection [id: 0x523effad, /127.0.0.1:38954 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.687 [DEBUG - New I/O worker #2493] (Negotiator.java:764) Authenticated connection [id: 0x26a370c9, /127.0.0.1:40918 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.692 [DEBUG - New I/O worker #2342] (Negotiator.java:764) Authenticated connection [id: 0x99798427, /127.0.0.1:54312 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.695 [DEBUG - New I/O worker #2287] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.697 [DEBUG - New I/O worker #2523] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.699 [DEBUG - New I/O worker #2310] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.701 [DEBUG - New I/O worker #2647] (Negotiator.java:764) Authenticated connection [id: 0x7624610b, /127.0.0.1:40948 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.704 [DEBUG - New I/O worker #2272] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.706 [DEBUG - New I/O worker #2525] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.710 [DEBUG - New I/O worker #2536] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.710 [DEBUG - New I/O worker #2494] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.712 [DEBUG - New I/O worker #2523] (Negotiator.java:764) Authenticated connection [id: 0xa512cf8e, /127.0.0.1:54324 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.713 [DEBUG - New I/O worker #2294] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.715 [DEBUG - New I/O worker #2242] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:42.715 [DEBUG - New I/O worker #2242] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:42.715 [DEBUG - New I/O worker #2242] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:42.715 [DEBUG - New I/O worker #2242] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:42.715 [DEBUG - New I/O worker #2242] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:42.716 [DEBUG - pool-3-thread-36] (Deferred.java:1330) callback=disconnect callback@1064332804 returned Deferred@985170343(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1058579183 after disconnect callback@1064332804), errback=passthrough -> (continuation of Deferred@1058579183 after disconnect callback@1064332804)), so the following Deferred is getting paused: Deferred@1058579183(state=PAUSED, result=Deferred@985170343, callback=, errback=) 00:27:42.716 [DEBUG - New I/O worker #2259] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:42.717 [DEBUG - New I/O worker #2341] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:42.717 [DEBUG - New I/O worker #2242] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:42.719 [DEBUG - New I/O worker #2258] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.726 [DEBUG - New I/O worker #2347] (Negotiator.java:764) Authenticated connection [id: 0xaf11fa7a, /127.0.0.1:54282 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.727 [DEBUG - New I/O worker #2267] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.727 [DEBUG - New I/O worker #2253] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.728 [DEBUG - New I/O worker #2311] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.729 [DEBUG - New I/O worker #2248] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.733 [DEBUG - New I/O worker #2268] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.735 [DEBUG - New I/O worker #2258] (Negotiator.java:764) Authenticated connection [id: 0x07e5296e, /127.0.0.1:54342 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.740 [DEBUG - New I/O worker #2261] (Negotiator.java:764) Authenticated connection [id: 0xea7b63f8, /127.0.0.1:54294 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.741 [DEBUG - New I/O worker #2350] (Negotiator.java:764) Authenticated connection [id: 0xfe1734ec, /127.0.0.1:54276 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.742 [DEBUG - New I/O worker #2490] (Negotiator.java:764) Authenticated connection [id: 0x5b4f5839, /127.0.0.1:54318 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.747 [DEBUG - New I/O worker #2272] (Negotiator.java:764) Authenticated connection [id: 0x2413dfd4, /127.0.0.1:54330 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.749 [DEBUG - New I/O worker #2268] (Negotiator.java:764) Authenticated connection [id: 0x48fd8135, /127.0.0.1:54336 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.750 [DEBUG - New I/O worker #2270] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.755 [DEBUG - New I/O worker #2323] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.757 [DEBUG - New I/O worker #2491] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.758 [DEBUG - New I/O worker #2330] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.762 [DEBUG - New I/O worker #2317] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:42.778 [DEBUG - pool-3-thread-53] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:42.779 [DEBUG - New I/O worker #2276] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:42.779 [DEBUG - New I/O worker #2251] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:42.780 [DEBUG - New I/O worker #2257] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.782 [DEBUG - New I/O worker #2343] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.782 [DEBUG - New I/O worker #2306] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.783 [DEBUG - New I/O worker #2547] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.784 [DEBUG - New I/O worker #2266] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.784 [DEBUG - New I/O worker #2482] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.784 [DEBUG - New I/O worker #2403] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.785 [DEBUG - New I/O worker #2307] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.786 [DEBUG - New I/O worker #2308] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.791 [DEBUG - New I/O worker #2485] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.791 [DEBUG - New I/O worker #2254] (Negotiator.java:764) Authenticated connection [id: 0x171cc080, /127.0.0.1:38948 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.792 [DEBUG - New I/O worker #2305] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.793 [DEBUG - New I/O worker #2246] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.794 [DEBUG - New I/O worker #2537] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.795 [DEBUG - New I/O worker #2298] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.797 [DEBUG - New I/O worker #2291] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.799 [DEBUG - New I/O worker #2285] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.800 [DEBUG - New I/O worker #2425] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.799 [DEBUG - New I/O worker #2304] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.801 [DEBUG - New I/O worker #2321] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.802 [DEBUG - New I/O worker #2320] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.802 [DEBUG - New I/O worker #2243] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:42.803 [DEBUG - New I/O worker #2300] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:42.803 [DEBUG - New I/O worker #2300] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=319), Traces: [0ms] refreshing cache from master, [135ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [135ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [135ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [207ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [217ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [319ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:42.804 [DEBUG - New I/O worker #2300] (Deferred.java:1330) callback=retry RPC@1953574972 returned Deferred@1884207722(state=PENDING, result=null, callback=(continuation of Deferred@894188039 after retry RPC@1953574972), errback=(continuation of Deferred@894188039 after retry RPC@1953574972)), so the following Deferred is getting paused: Deferred@894188039(state=PAUSED, result=Deferred@1884207722, callback=org.apache.kudu.client.AsyncKuduClient$6@793e03b5 -> wakeup thread pool-3-thread-79, errback=passthrough -> wakeup thread pool-3-thread-79) 00:27:42.806 [DEBUG - New I/O worker #2487] (Negotiator.java:764) Authenticated connection [id: 0x56952830, /127.0.0.1:40942 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.807 [DEBUG - New I/O worker #2494] (Negotiator.java:764) Authenticated connection [id: 0x498441b9, /127.0.0.1:40966 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.808 [DEBUG - New I/O worker #2351] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.808 [DEBUG - New I/O worker #2654] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.808 [DEBUG - New I/O worker #2318] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.810 [DEBUG - New I/O worker #2283] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.811 [DEBUG - New I/O worker #2520] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.811 [DEBUG - New I/O worker #2582] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.811 [DEBUG - New I/O worker #2437] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.826 [DEBUG - New I/O worker #2266] (Negotiator.java:764) Authenticated connection [id: 0xf840a7eb, /127.0.0.1:54390 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.826 [DEBUG - New I/O worker #2293] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.844 [DEBUG - New I/O worker #2287] (Negotiator.java:764) Authenticated connection [id: 0x61f13fe9, /127.0.0.1:38930 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.855 [DEBUG - New I/O worker #2246] (Negotiator.java:764) Authenticated connection [id: 0x564de3d6, /127.0.0.1:38984 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.861 [DEBUG - New I/O worker #2536] (Negotiator.java:764) Authenticated connection [id: 0x722fb8db, /127.0.0.1:40960 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.862 [DEBUG - New I/O worker #2263] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.862 [DEBUG - New I/O worker #2291] (Negotiator.java:764) Authenticated connection [id: 0x86038321, /127.0.0.1:39026 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.863 [DEBUG - New I/O worker #2295] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.866 [DEBUG - New I/O worker #2332] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.866 [DEBUG - New I/O worker #2338] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.867 [DEBUG - New I/O worker #2244] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.868 [DEBUG - New I/O worker #2337] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.866 [DEBUG - New I/O worker #2340] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.866 [DEBUG - New I/O worker #2329] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.874 [DEBUG - New I/O worker #2506] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.884 [DEBUG - New I/O worker #2293] (Negotiator.java:764) Authenticated connection [id: 0xaa65c2c0, /127.0.0.1:54408 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.885 [DEBUG - New I/O worker #2510] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.891 [DEBUG - New I/O worker #2425] (Negotiator.java:764) Authenticated connection [id: 0x73a125f8, /127.0.0.1:41020 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.891 [DEBUG - New I/O worker #2437] (Negotiator.java:764) Authenticated connection [id: 0xf5082447, /127.0.0.1:40972 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.901 [DEBUG - New I/O worker #2548] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.902 [DEBUG - New I/O worker #2521] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.903 [DEBUG - New I/O worker #2481] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:42.918 [DEBUG - New I/O worker #2485] (Negotiator.java:764) Authenticated connection [id: 0x76a89c64, /127.0.0.1:41032 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.919 [DEBUG - New I/O worker #2482] (Negotiator.java:764) Authenticated connection [id: 0x9ac103c8, /127.0.0.1:41008 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.921 [DEBUG - New I/O worker #2343] (Negotiator.java:764) Authenticated connection [id: 0x2596ca8c, /127.0.0.1:54384 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.935 [DEBUG - New I/O worker #2582] (Negotiator.java:764) Authenticated connection [id: 0x5ba42ddd, /127.0.0.1:40996 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.939 [DEBUG - New I/O worker #2521] (Negotiator.java:764) Authenticated connection [id: 0xc3b85376, /127.0.0.1:41044 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.946 [DEBUG - New I/O worker #2252] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.957 [DEBUG - New I/O worker #2520] (Negotiator.java:764) Authenticated connection [id: 0x261bc84b, /127.0.0.1:41002 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.957 [DEBUG - New I/O worker #2510] (Negotiator.java:764) Authenticated connection [id: 0x95ed80e1, /127.0.0.1:41062 => /127.22.19.253:37879] using SASL/PLAIN 00:27:43.000 [DEBUG - New I/O worker #2283] (Negotiator.java:764) Authenticated connection [id: 0x325b3d47, /127.0.0.1:38924 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.002 [DEBUG - New I/O worker #2263] (Negotiator.java:764) Authenticated connection [id: 0xa85414af, /127.0.0.1:39062 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.004 [DEBUG - New I/O worker #2285] (Negotiator.java:764) Authenticated connection [id: 0xcd4b4694, /127.0.0.1:39050 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.006 [DEBUG - New I/O worker #2337] (Negotiator.java:764) Authenticated connection [id: 0xfb4830e6, /127.0.0.1:54422 => /127.22.19.254:44849] using SASL/PLAIN 00:27:43.008 [DEBUG - New I/O worker #2294] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.008 [DEBUG - New I/O worker #2294] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.008 [DEBUG - New I/O worker #2294] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.008 [DEBUG - New I/O worker #2294] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.009 [DEBUG - New I/O worker #2294] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.009 [DEBUG - pool-3-thread-27] (Deferred.java:1330) callback=disconnect callback@2034770198 returned Deferred@988949411(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@456539280 after disconnect callback@2034770198), errback=passthrough -> (continuation of Deferred@456539280 after disconnect callback@2034770198)), so the following Deferred is getting paused: Deferred@456539280(state=PAUSED, result=Deferred@988949411, callback=, errback=) 00:27:43.010 [DEBUG - New I/O worker #2344] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.010 [DEBUG - New I/O worker #2476] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.011 [DEBUG - New I/O worker #2291] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.011 [DEBUG - New I/O worker #2291] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=538), Traces: [0ms] refreshing cache from master, [283ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [283ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [283ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [487ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [538ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.012 [DEBUG - New I/O worker #2291] (Deferred.java:1330) callback=retry RPC@2013097727 returned Deferred@1942152408(state=PENDING, result=null, callback=(continuation of Deferred@918578372 after retry RPC@2013097727), errback=(continuation of Deferred@918578372 after retry RPC@2013097727)), so the following Deferred is getting paused: Deferred@918578372(state=PAUSED, result=Deferred@1942152408, callback=org.apache.kudu.client.AsyncKuduClient$6@1baaea7b -> wakeup thread pool-3-thread-55, errback=passthrough -> wakeup thread pool-3-thread-55) 00:27:42.997 [DEBUG - New I/O worker #2304] (Negotiator.java:764) Authenticated connection [id: 0xa28a1f8d, /127.0.0.1:39032 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.997 [DEBUG - New I/O worker #2256] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:42.996 [DEBUG - New I/O worker #2427] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:42.996 [DEBUG - New I/O worker #2339] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:42.994 [DEBUG - New I/O worker #2298] (Negotiator.java:764) Authenticated connection [id: 0xc9feac67, /127.0.0.1:39012 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.993 [DEBUG - New I/O worker #2318] (Negotiator.java:764) Authenticated connection [id: 0x894bdacd, /127.0.0.1:38978 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.993 [DEBUG - New I/O worker #2242] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:42.991 [DEBUG - New I/O worker #2270] (Negotiator.java:764) Authenticated connection [id: 0xa76ed704, /127.0.0.1:54348 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.990 [DEBUG - New I/O worker #2349] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.988 [DEBUG - New I/O worker #2328] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.254 00:27:42.987 [DEBUG - New I/O worker #2525] (Negotiator.java:764) Authenticated connection [id: 0xb39b1407, /127.0.0.1:40954 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.987 [DEBUG - New I/O worker #2534] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.253 00:27:43.020 [DEBUG - New I/O worker #2491] (Negotiator.java:764) Authenticated connection [id: 0x0aa4cdc3, /127.0.0.1:54366 => /127.22.19.254:44849] using SASL/PLAIN 00:27:43.028 [DEBUG - New I/O worker #2297] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.985 [DEBUG - New I/O worker #2329] (Negotiator.java:764) Authenticated connection [id: 0x0d7d139a, /127.0.0.1:54414 => /127.22.19.254:44849] using SASL/PLAIN 00:27:43.030 [DEBUG - New I/O worker #2537] (Negotiator.java:764) Authenticated connection [id: 0xbf5c1583, /127.0.0.1:54396 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.980 [DEBUG - New I/O worker #2351] (Negotiator.java:764) Authenticated connection [id: 0x35eb2c71, /127.0.0.1:54402 => /127.22.19.254:44849] using SASL/PLAIN 00:27:42.979 [DEBUG - New I/O worker #2321] (Negotiator.java:764) Authenticated connection [id: 0xd865b913, /127.0.0.1:39056 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.034 [DEBUG - New I/O worker #2506] (Negotiator.java:764) Authenticated connection [id: 0x640379f4, /127.0.0.1:41054 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.979 [DEBUG - New I/O worker #2310] (Negotiator.java:764) Authenticated connection [id: 0x0ae8eded, /127.0.0.1:38942 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.977 [DEBUG - New I/O worker #2548] (Negotiator.java:764) Authenticated connection [id: 0xf15227ff, /127.0.0.1:41056 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.977 [DEBUG - New I/O worker #2247] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.975 [DEBUG - New I/O worker #2305] (Negotiator.java:764) Authenticated connection [id: 0xd3641762, /127.0.0.1:38936 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.975 [DEBUG - New I/O worker #2307] (Negotiator.java:764) Authenticated connection [id: 0x097676a8, /127.0.0.1:38960 => /127.22.19.252:32833] using SASL/PLAIN 00:27:42.973 [DEBUG - New I/O worker #2547] (Negotiator.java:764) Authenticated connection [id: 0x61e29690, /127.0.0.1:40990 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.973 [DEBUG - New I/O worker #2340] (Negotiator.java:764) Authenticated connection [id: 0x0bcd51c5, /127.0.0.1:41038 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.971 [DEBUG - New I/O worker #2481] (Negotiator.java:764) Authenticated connection [id: 0xc0389bf7, /127.0.0.1:41014 => /127.22.19.253:37879] using SASL/PLAIN 00:27:42.967 [DEBUG - New I/O worker #2403] (Negotiator.java:764) Authenticated connection [id: 0x93d3cd76, /127.0.0.1:40978 => /127.22.19.253:37879] using SASL/PLAIN 00:27:43.040 [DEBUG - New I/O worker #2249] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:42.964 [DEBUG - New I/O worker #2654] (Negotiator.java:764) Authenticated connection [id: 0xbfada841, /127.0.0.1:41026 => /127.22.19.253:37879] using SASL/PLAIN 00:27:43.035 [DEBUG - New I/O worker #2312] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:43.013 [DEBUG - New I/O worker #2323] (Negotiator.java:764) Authenticated connection [id: 0x13622aa6, /127.0.0.1:54354 => /127.22.19.254:44849] using SASL/PLAIN 00:27:43.011 [DEBUG - New I/O worker #2294] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.008 [DEBUG - New I/O worker #2309] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.043 [DEBUG - New I/O worker #2309] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.044 [DEBUG - New I/O worker #2309] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.044 [DEBUG - New I/O worker #2309] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.044 [DEBUG - New I/O worker #2309] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.045 [DEBUG - pool-3-thread-66] (Deferred.java:1330) callback=disconnect callback@567528411 returned Deferred@612762054(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@798298477 after disconnect callback@567528411), errback=passthrough -> (continuation of Deferred@798298477 after disconnect callback@567528411)), so the following Deferred is getting paused: Deferred@798298477(state=PAUSED, result=Deferred@612762054, callback=, errback=) 00:27:43.045 [DEBUG - New I/O worker #2295] (Negotiator.java:764) Authenticated connection [id: 0x61ea73c2, /127.0.0.1:39074 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.045 [DEBUG - New I/O worker #2486] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.044 [DEBUG - New I/O worker #2300] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.045 [DEBUG - New I/O worker #2348] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.047 [DEBUG - New I/O worker #2309] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.048 [DEBUG - New I/O worker #2265] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.048 [DEBUG - New I/O worker #2265] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=970), Traces: [0ms] refreshing cache from master, [20ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [20ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [27ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [449ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [484ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [970ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.049 [DEBUG - New I/O worker #2265] (Deferred.java:1330) callback=retry RPC@61408954 returned Deferred@127308311(state=PENDING, result=null, callback=(continuation of Deferred@1116998785 after retry RPC@61408954), errback=(continuation of Deferred@1116998785 after retry RPC@61408954)), so the following Deferred is getting paused: Deferred@1116998785(state=PAUSED, result=Deferred@127308311, callback=org.apache.kudu.client.AsyncKuduClient$6@384b7075 -> wakeup thread pool-3-thread-30, errback=passthrough -> wakeup thread pool-3-thread-30) 00:27:43.050 [DEBUG - New I/O worker #2289] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.050 [DEBUG - New I/O worker #2289] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1038), Traces: [0ms] refreshing cache from master, [27ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [31ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [35ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [546ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [549ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1038ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.050 [DEBUG - New I/O worker #2289] (Deferred.java:1330) callback=retry RPC@1211567899 returned Deferred@1275626181(state=PENDING, result=null, callback=(continuation of Deferred@151651616 after retry RPC@1211567899), errback=(continuation of Deferred@151651616 after retry RPC@1211567899)), so the following Deferred is getting paused: Deferred@151651616(state=PAUSED, result=Deferred@1275626181, callback=org.apache.kudu.client.AsyncKuduClient$6@48f6f8f5 -> wakeup thread pool-3-thread-48, errback=passthrough -> wakeup thread pool-3-thread-48) 00:27:43.052 [DEBUG - New I/O worker #2271] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.052 [DEBUG - New I/O worker #2271] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1050), Traces: [0ms] refreshing cache from master, [44ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [51ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [64ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [316ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [548ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1051ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.052 [DEBUG - New I/O worker #2271] (Deferred.java:1330) callback=retry RPC@929998859 returned Deferred@860961342(state=PENDING, result=null, callback=(continuation of Deferred@1985192496 after retry RPC@929998859), errback=(continuation of Deferred@1985192496 after retry RPC@929998859)), so the following Deferred is getting paused: Deferred@1985192496(state=PAUSED, result=Deferred@860961342, callback=org.apache.kudu.client.AsyncKuduClient$6@503be95 -> wakeup thread pool-3-thread-47, errback=passthrough -> wakeup thread pool-3-thread-47) 00:27:43.052 [DEBUG - New I/O worker #2247] (Negotiator.java:764) Authenticated connection [id: 0xf89f2523, /127.0.0.1:39008 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.053 [DEBUG - New I/O worker #2309] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.054 [DEBUG - New I/O worker #2281] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:43.055 [DEBUG - New I/O worker #2294] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.059 [DEBUG - New I/O worker #2246] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.059 [DEBUG - New I/O worker #2246] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=575), Traces: [0ms] refreshing cache from master, [217ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [218ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [218ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [267ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [462ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [575ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.060 [DEBUG - New I/O worker #2246] (Deferred.java:1330) callback=retry RPC@1409083766 returned Deferred@1472427985(state=PENDING, result=null, callback=(continuation of Deferred@314696525 after retry RPC@1409083766), errback=(continuation of Deferred@314696525 after retry RPC@1409083766)), so the following Deferred is getting paused: Deferred@314696525(state=PAUSED, result=Deferred@1472427985, callback=org.apache.kudu.client.AsyncKuduClient$6@179a3ba0 -> wakeup thread pool-3-thread-54, errback=passthrough -> wakeup thread pool-3-thread-54) 00:27:43.060 [DEBUG - New I/O worker #2328] (Negotiator.java:764) Authenticated connection [id: 0x2f8c21b3, /127.0.0.1:54372 => /127.22.19.254:44849] using SASL/PLAIN 00:27:43.064 [DEBUG - New I/O worker #2330] (Negotiator.java:764) Authenticated connection [id: 0x608b42c0, /127.0.0.1:54360 => /127.22.19.254:44849] using SASL/PLAIN 00:27:43.064 [DEBUG - New I/O worker #2275] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.22.19.252 00:27:43.062 [DEBUG - New I/O worker #2291] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.064 [DEBUG - New I/O worker #2285] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.066 [DEBUG - New I/O worker #2285] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=597), Traces: [0ms] refreshing cache from master, [318ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [318ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [318ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [569ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [580ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [596ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.066 [DEBUG - New I/O worker #2285] (Deferred.java:1330) callback=retry RPC@421854870 returned Deferred@488339818(state=PENDING, result=null, callback=(continuation of Deferred@1478096045 after retry RPC@421854870), errback=(continuation of Deferred@1478096045 after retry RPC@421854870)), so the following Deferred is getting paused: Deferred@1478096045(state=PAUSED, result=Deferred@488339818, callback=org.apache.kudu.client.AsyncKuduClient$6@57e54e76 -> wakeup thread pool-3-thread-28, errback=passthrough -> wakeup thread pool-3-thread-28) 00:27:43.065 [DEBUG - New I/O worker #2349] (Negotiator.java:764) Authenticated connection [id: 0x8466abf9, /127.0.0.1:54378 => /127.22.19.254:44849] using SASL/PLAIN 00:27:43.066 [DEBUG - New I/O worker #2534] (Negotiator.java:764) Authenticated connection [id: 0x1f5d8c01, /127.0.0.1:40986 => /127.22.19.253:37879] using SASL/PLAIN 00:27:43.070 [DEBUG - New I/O worker #2484] (Negotiator.java:764) Authenticated connection [id: 0xedf7ae64, /127.0.0.1:40912 => /127.22.19.253:37879] using SASL/PLAIN 00:27:43.074 [DEBUG - New I/O worker #2257] (Negotiator.java:764) Authenticated connection [id: 0x48126198, /127.0.0.1:38996 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.074 [DEBUG - New I/O worker #2283] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.075 [DEBUG - New I/O worker #2283] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=747), Traces: [0ms] refreshing cache from master, [245ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [245ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [245ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [280ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [475ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [747ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.075 [DEBUG - New I/O worker #2244] (Negotiator.java:764) Authenticated connection [id: 0x7a6d56e9, /127.0.0.1:39002 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.076 [DEBUG - New I/O worker #2252] (Negotiator.java:764) Authenticated connection [id: 0x26e9346d, /127.0.0.1:38990 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.077 [DEBUG - New I/O worker #2243] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.077 [DEBUG - New I/O worker #2243] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.077 [DEBUG - New I/O worker #2243] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.078 [DEBUG - New I/O worker #2243] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.078 [DEBUG - New I/O worker #2243] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.080 [DEBUG - New I/O worker #2303] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.080 [DEBUG - New I/O worker #2286] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.080 [DEBUG - New I/O worker #2286] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.080 [DEBUG - New I/O worker #2286] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.080 [DEBUG - New I/O worker #2267] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.081 [DEBUG - New I/O worker #2267] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.081 [DEBUG - New I/O worker #2267] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.081 [DEBUG - New I/O worker #2267] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.081 [DEBUG - New I/O worker #2267] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.080 [DEBUG - New I/O worker #2274] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.091 [DEBUG - New I/O worker #2338] (Negotiator.java:764) Authenticated connection [id: 0x124ea669, /127.0.0.1:54432 => /127.22.19.254:44849] using SASL/PLAIN 00:27:43.091 [DEBUG - New I/O worker #2308] (Negotiator.java:764) Authenticated connection [id: 0x813d690b, /127.0.0.1:38972 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.090 [DEBUG - New I/O worker #2283] (Deferred.java:1330) callback=retry RPC@2015287014 returned Deferred@2082561341(state=PENDING, result=null, callback=(continuation of Deferred@958651613 after retry RPC@2015287014), errback=(continuation of Deferred@958651613 after retry RPC@2015287014)), so the following Deferred is getting paused: Deferred@958651613(state=PAUSED, result=Deferred@2082561341, callback=org.apache.kudu.client.AsyncKuduClient$6@677eeea -> wakeup thread pool-3-thread-49, errback=passthrough -> wakeup thread pool-3-thread-49) 00:27:43.090 [DEBUG - New I/O worker #2332] (Negotiator.java:764) Authenticated connection [id: 0xad6962fd, /127.0.0.1:54424 => /127.22.19.254:44849] using SASL/PLAIN 00:27:43.089 [DEBUG - New I/O worker #2535] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.099 [DEBUG - New I/O worker #2311] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.099 [DEBUG - New I/O worker #2311] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.100 [DEBUG - New I/O worker #2311] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.089 [DEBUG - New I/O worker #2324] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.100 [DEBUG - New I/O worker #2248] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.100 [DEBUG - New I/O worker #2248] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.101 [DEBUG - New I/O worker #2248] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.101 [DEBUG - New I/O worker #2248] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.101 [DEBUG - New I/O worker #2248] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.102 [DEBUG - pool-3-thread-6] (Deferred.java:1330) callback=disconnect callback@1327006669 returned Deferred@274371022(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1387985406 after disconnect callback@1327006669), errback=passthrough -> (continuation of Deferred@1387985406 after disconnect callback@1327006669)), so the following Deferred is getting paused: Deferred@1387985406(state=PAUSED, result=Deferred@274371022, callback=, errback=) 00:27:43.102 [DEBUG - New I/O worker #2255] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.102 [DEBUG - New I/O worker #2262] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.088 [DEBUG - New I/O worker #2267] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.088 [DEBUG - New I/O worker #2334] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.088 [DEBUG - pool-3-thread-9] (Deferred.java:1330) callback=disconnect callback@1484692099 returned Deferred@1904041036(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@12487789 after disconnect callback@1484692099), errback=passthrough -> (continuation of Deferred@12487789 after disconnect callback@1484692099)), so the following Deferred is getting paused: Deferred@12487789(state=PAUSED, result=Deferred@1904041036, callback=, errback=) 00:27:43.088 [DEBUG - pool-3-thread-38] (Deferred.java:1330) callback=disconnect callback@1677217161 returned Deferred@1261426039(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1305599233 after disconnect callback@1677217161), errback=passthrough -> (continuation of Deferred@1305599233 after disconnect callback@1677217161)), so the following Deferred is getting paused: Deferred@1305599233(state=PAUSED, result=Deferred@1261426039, callback=, errback=) 00:27:43.081 [DEBUG - New I/O worker #2281] (Negotiator.java:764) Authenticated connection [id: 0x655e5303, /127.0.0.1:39076 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.080 [DEBUG - New I/O worker #2286] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.104 [DEBUG - New I/O worker #2286] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.080 [DEBUG - New I/O worker #2303] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1078), Traces: [0ms] refreshing cache from master, [40ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [45ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [45ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [332ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [543ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [1079ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.104 [DEBUG - New I/O worker #2303] (Deferred.java:1330) callback=retry RPC@525372461 returned Deferred@460266271(state=PENDING, result=null, callback=(continuation of Deferred@1584236054 after retry RPC@525372461), errback=(continuation of Deferred@1584236054 after retry RPC@525372461)), so the following Deferred is getting paused: Deferred@1584236054(state=PAUSED, result=Deferred@460266271, callback=org.apache.kudu.client.AsyncKuduClient$6@74d381bc -> wakeup thread pool-3-thread-59, errback=passthrough -> wakeup thread pool-3-thread-59) 00:27:43.105 [DEBUG - pool-3-thread-4] (Deferred.java:1330) callback=disconnect callback@660070598 returned Deferred@390668242(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@343361172 after disconnect callback@660070598), errback=passthrough -> (continuation of Deferred@343361172 after disconnect callback@660070598)), so the following Deferred is getting paused: Deferred@343361172(state=PAUSED, result=Deferred@390668242, callback=, errback=) 00:27:43.080 [DEBUG - New I/O worker #2243] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.106 [DEBUG - New I/O worker #2290] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.106 [DEBUG - New I/O worker #2353] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.107 [DEBUG - New I/O worker #2286] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.103 [DEBUG - New I/O worker #2248] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.102 [DEBUG - New I/O worker #2306] (Negotiator.java:764) Authenticated connection [id: 0x5ee37d32, /127.0.0.1:38966 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.100 [DEBUG - New I/O worker #2311] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.108 [DEBUG - New I/O worker #2311] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.100 [DEBUG - New I/O worker #2253] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.109 [DEBUG - New I/O worker #2253] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.109 [DEBUG - New I/O worker #2253] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.109 [DEBUG - New I/O worker #2253] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.109 [DEBUG - New I/O worker #2253] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.110 [DEBUG - New I/O worker #2313] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.110 [DEBUG - New I/O worker #2313] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.110 [DEBUG - New I/O worker #2313] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.110 [DEBUG - New I/O worker #2313] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.111 [DEBUG - New I/O worker #2313] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.111 [DEBUG - pool-3-thread-62] (Deferred.java:1330) callback=disconnect callback@1633064204 returned Deferred@1136058401(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1322747716 after disconnect callback@1633064204), errback=passthrough -> (continuation of Deferred@1322747716 after disconnect callback@1633064204)), so the following Deferred is getting paused: Deferred@1322747716(state=PAUSED, result=Deferred@1136058401, callback=, errback=) 00:27:43.112 [DEBUG - pool-3-thread-69] (Deferred.java:1330) callback=disconnect callback@1636249066 returned Deferred@1555457912(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@660693579 after disconnect callback@1636249066), errback=passthrough -> (continuation of Deferred@660693579 after disconnect callback@1636249066)), so the following Deferred is getting paused: Deferred@660693579(state=PAUSED, result=Deferred@1555457912, callback=, errback=) 00:27:43.112 [DEBUG - New I/O worker #2264] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.112 [DEBUG - New I/O worker #2527] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.113 [DEBUG - New I/O worker #2423] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.113 [DEBUG - New I/O worker #2424] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.114 [DEBUG - pool-3-thread-21] (Deferred.java:1330) callback=disconnect callback@75088819 returned Deferred@1975649665(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@454943650 after disconnect callback@75088819), errback=passthrough -> (continuation of Deferred@454943650 after disconnect callback@75088819)), so the following Deferred is getting paused: Deferred@454943650(state=PAUSED, result=Deferred@1975649665, callback=, errback=) 00:27:43.114 [DEBUG - New I/O worker #2333] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.115 [DEBUG - New I/O worker #2529] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.115 [DEBUG - New I/O worker #2300] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.115 [DEBUG - New I/O worker #2300] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.116 [DEBUG - New I/O worker #2300] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.116 [DEBUG - New I/O worker #2300] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.116 [DEBUG - New I/O worker #2300] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.117 [DEBUG - pool-3-thread-79] (Deferred.java:1330) callback=disconnect callback@1229611503 returned Deferred@2006616411(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1149490549 after disconnect callback@1229611503), errback=passthrough -> (continuation of Deferred@1149490549 after disconnect callback@1229611503)), so the following Deferred is getting paused: Deferred@1149490549(state=PAUSED, result=Deferred@2006616411, callback=, errback=) 00:27:43.117 [DEBUG - New I/O worker #2352] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.117 [DEBUG - New I/O worker #2475] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.107 [DEBUG - New I/O worker #2312] (Negotiator.java:764) Authenticated connection [id: 0x9f204814, /127.0.0.1:39038 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.118 [DEBUG - New I/O worker #2321] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.118 [DEBUG - New I/O worker #2321] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=647), Traces: [0ms] refreshing cache from master, [315ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [316ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [316ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [453ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [586ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [646ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.118 [DEBUG - New I/O worker #2321] (Deferred.java:1330) callback=retry RPC@832854430 returned Deferred@899402610(state=PENDING, result=null, callback=(continuation of Deferred@1889096613 after retry RPC@832854430), errback=(continuation of Deferred@1889096613 after retry RPC@832854430)), so the following Deferred is getting paused: Deferred@1889096613(state=PAUSED, result=Deferred@899402610, callback=org.apache.kudu.client.AsyncKuduClient$6@52d89e04 -> wakeup thread pool-3-thread-77, errback=passthrough -> wakeup thread pool-3-thread-77) 00:27:43.119 [DEBUG - New I/O worker #2300] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.119 [DEBUG - New I/O worker #2313] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.119 [DEBUG - New I/O worker #2253] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.120 [DEBUG - New I/O worker #2311] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.120 [DEBUG - New I/O worker #2295] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.120 [DEBUG - New I/O worker #2295] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=791), Traces: [0ms] refreshing cache from master, [501ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [501ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [502ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [727ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [773ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [791ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.120 [DEBUG - New I/O worker #2295] (Deferred.java:1330) callback=retry RPC@569248179 returned Deferred@634489174(state=PENDING, result=null, callback=(continuation of Deferred@1624441736 after retry RPC@569248179), errback=(continuation of Deferred@1624441736 after retry RPC@569248179)), so the following Deferred is getting paused: Deferred@1624441736(state=PAUSED, result=Deferred@634489174, callback=org.apache.kudu.client.AsyncKuduClient$6@be566 -> wakeup thread pool-3-thread-1, errback=passthrough -> wakeup thread pool-3-thread-1) 00:27:43.121 [DEBUG - New I/O worker #2298] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.121 [DEBUG - New I/O worker #2298] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=647), Traces: [0ms] refreshing cache from master, [278ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [278ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [278ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [574ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [596ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [647ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.122 [DEBUG - New I/O worker #2298] (Deferred.java:1330) callback=retry RPC@184581294 returned Deferred@255850774(state=PENDING, result=null, callback=(continuation of Deferred@1245542037 after retry RPC@184581294), errback=(continuation of Deferred@1245542037 after retry RPC@184581294)), so the following Deferred is getting paused: Deferred@1245542037(state=PAUSED, result=Deferred@255850774, callback=org.apache.kudu.client.AsyncKuduClient$6@73d08916 -> wakeup thread pool-3-thread-35, errback=passthrough -> wakeup thread pool-3-thread-35) 00:27:43.121 [DEBUG - New I/O worker #2320] (Negotiator.java:764) Authenticated connection [id: 0x31dd703e, /127.0.0.1:39020 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.123 [DEBUG - New I/O worker #2304] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.123 [DEBUG - New I/O worker #2283] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.123 [DEBUG - New I/O worker #2310] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.123 [DEBUG - New I/O worker #2246] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.124 [DEBUG - New I/O worker #2304] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=647), Traces: [0ms] refreshing cache from master, [282ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [282ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [283ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [447ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [593ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [645ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.124 [DEBUG - New I/O worker #2310] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=639), Traces: [0ms] refreshing cache from master, [122ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [123ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [123ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [205ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [216ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [638ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.124 [DEBUG - New I/O worker #2304] (Deferred.java:1330) callback=retry RPC@107550420 returned Deferred@39224001(state=PENDING, result=null, callback=(continuation of Deferred@1196694767 after retry RPC@107550420), errback=(continuation of Deferred@1196694767 after retry RPC@107550420)), so the following Deferred is getting paused: Deferred@1196694767(state=PAUSED, result=Deferred@39224001, callback=org.apache.kudu.client.AsyncKuduClient$6@38c886d5 -> wakeup thread pool-3-thread-56, errback=passthrough -> wakeup thread pool-3-thread-56) 00:27:43.124 [DEBUG - New I/O worker #2310] (Deferred.java:1330) callback=retry RPC@556887377 returned Deferred@621735343(state=PENDING, result=null, callback=(continuation of Deferred@1611425642 after retry RPC@556887377), errback=(continuation of Deferred@1611425642 after retry RPC@556887377)), so the following Deferred is getting paused: Deferred@1611425642(state=PAUSED, result=Deferred@621735343, callback=org.apache.kudu.client.AsyncKuduClient$6@7bd7d5ca -> wakeup thread pool-3-thread-71, errback=passthrough -> wakeup thread pool-3-thread-71) 00:27:43.125 [DEBUG - New I/O worker #2291] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.125 [DEBUG - New I/O worker #2291] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.125 [DEBUG - New I/O worker #2291] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.125 [DEBUG - New I/O worker #2291] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.125 [DEBUG - New I/O worker #2291] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.126 [DEBUG - pool-3-thread-55] (Deferred.java:1330) callback=disconnect callback@392234864 returned Deferred@824061195(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@999941930 after disconnect callback@392234864), errback=passthrough -> (continuation of Deferred@999941930 after disconnect callback@392234864)), so the following Deferred is getting paused: Deferred@999941930(state=PAUSED, result=Deferred@824061195, callback=, errback=) 00:27:43.126 [DEBUG - New I/O worker #2306] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.125 [DEBUG - New I/O worker #2285] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.127 [DEBUG - New I/O worker #2244] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.127 [DEBUG - New I/O worker #2244] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=651), Traces: [0ms] refreshing cache from master, [272ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [272ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [273ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [554ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [573ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [652ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.126 [DEBUG - New I/O worker #2306] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=642), Traces: [0ms] refreshing cache from master, [166ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [166ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [166ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [238ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [409ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [642ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.128 [DEBUG - New I/O worker #2306] (Deferred.java:1330) callback=retry RPC@1417105476 returned Deferred@1346948626(state=PENDING, result=null, callback=(continuation of Deferred@357194367 after retry RPC@1417105476), errback=(continuation of Deferred@357194367 after retry RPC@1417105476)), so the following Deferred is getting paused: Deferred@357194367(state=PAUSED, result=Deferred@1346948626, callback=org.apache.kudu.client.AsyncKuduClient$6@4a0e6601 -> wakeup thread pool-3-thread-51, errback=passthrough -> wakeup thread pool-3-thread-51) 00:27:43.126 [DEBUG - New I/O worker #2520] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.126 [DEBUG - New I/O worker #2328] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.127 [DEBUG - New I/O worker #2244] (Deferred.java:1330) callback=retry RPC@1697552687 returned Deferred@1628508498(state=PENDING, result=null, callback=(continuation of Deferred@605262612 after retry RPC@1697552687), errback=(continuation of Deferred@605262612 after retry RPC@1697552687)), so the following Deferred is getting paused: Deferred@605262612(state=PAUSED, result=Deferred@1628508498, callback=org.apache.kudu.client.AsyncKuduClient$6@48bc5de -> wakeup thread pool-3-thread-11, errback=passthrough -> wakeup thread pool-3-thread-11) 00:27:43.127 [DEBUG - New I/O worker #2291] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.129 [DEBUG - New I/O worker #2254] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.129 [DEBUG - New I/O worker #2254] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=644), Traces: [0ms] refreshing cache from master, [127ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [127ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [127ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [201ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [261ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [644ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.129 [DEBUG - New I/O worker #2254] (Deferred.java:1330) callback=retry RPC@272208870 returned Deferred@335938873(state=PENDING, result=null, callback=(continuation of Deferred@1359256029 after retry RPC@272208870), errback=(continuation of Deferred@1359256029 after retry RPC@272208870)), so the following Deferred is getting paused: Deferred@1359256029(state=PAUSED, result=Deferred@335938873, callback=org.apache.kudu.client.AsyncKuduClient$6@3c652992 -> wakeup thread pool-3-thread-31, errback=passthrough -> wakeup thread pool-3-thread-31) 00:27:43.130 [DEBUG - New I/O worker #2257] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.130 [DEBUG - New I/O worker #2257] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=646), Traces: [0ms] refreshing cache from master, [232ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [232ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [233ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [256ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [415ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [646ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.130 [DEBUG - New I/O worker #2257] (Deferred.java:1330) callback=retry RPC@503193188 returned Deferred@432118019(state=PENDING, result=null, callback=(continuation of Deferred@1556288607 after retry RPC@503193188), errback=(continuation of Deferred@1556288607 after retry RPC@503193188)), so the following Deferred is getting paused: Deferred@1556288607(state=PAUSED, result=Deferred@432118019, callback=org.apache.kudu.client.AsyncKuduClient$6@35dfde -> wakeup thread Time-limited test, errback=passthrough -> wakeup thread Time-limited test) 00:27:43.131 [DEBUG - New I/O worker #2287] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.132 [DEBUG - New I/O worker #2287] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=646), Traces: [0ms] refreshing cache from master, [93ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [93ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [93ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [241ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [264ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [646ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.132 [DEBUG - New I/O worker #2287] (Deferred.java:1330) callback=retry RPC@528315660 returned Deferred@457368123(state=PENDING, result=null, callback=(continuation of Deferred@1581285175 after retry RPC@528315660), errback=(continuation of Deferred@1581285175 after retry RPC@528315660)), so the following Deferred is getting paused: Deferred@1581285175(state=PAUSED, result=Deferred@457368123, callback=org.apache.kudu.client.AsyncKuduClient$6@1b754565 -> wakeup thread pool-3-thread-58, errback=passthrough -> wakeup thread pool-3-thread-58) 00:27:43.139 [DEBUG - New I/O worker #2263] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.139 [DEBUG - New I/O worker #2263] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=809), Traces: [0ms] refreshing cache from master, [492ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [493ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [493ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [658ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [726ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [809ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.139 [DEBUG - New I/O worker #2263] (Deferred.java:1330) callback=retry RPC@2145627406 returned Deferred@2078029354(state=PENDING, result=null, callback=(continuation of Deferred@1054779189 after retry RPC@2145627406), errback=(continuation of Deferred@1054779189 after retry RPC@2145627406)), so the following Deferred is getting paused: Deferred@1054779189(state=PAUSED, result=Deferred@2078029354, callback=org.apache.kudu.client.AsyncKuduClient$6@6d43f339 -> wakeup thread pool-3-thread-20, errback=passthrough -> wakeup thread pool-3-thread-20) 00:27:43.139 [DEBUG - New I/O worker #2535] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.140 [DEBUG - New I/O worker #2318] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.140 [DEBUG - New I/O worker #2318] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=661), Traces: [0ms] refreshing cache from master, [215ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [216ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [217ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [237ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [545ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [661ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.140 [DEBUG - New I/O worker #2318] (Deferred.java:1330) callback=retry RPC@1390350161 returned Deferred@1457557970(state=PENDING, result=null, callback=(continuation of Deferred@333579626 after retry RPC@1390350161), errback=(continuation of Deferred@333579626 after retry RPC@1390350161)), so the following Deferred is getting paused: Deferred@333579626(state=PAUSED, result=Deferred@1457557970, callback=org.apache.kudu.client.AsyncKuduClient$6@8b4305e -> wakeup thread pool-3-thread-7, errback=passthrough -> wakeup thread pool-3-thread-7) 00:27:43.141 [DEBUG - New I/O worker #2297] (Negotiator.java:764) Authenticated connection [id: 0xab54bc8d, /127.0.0.1:39068 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.141 [DEBUG - New I/O worker #2321] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.143 [DEBUG - New I/O worker #2291] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.144 [DEBUG - New I/O worker #2249] (Negotiator.java:764) Authenticated connection [id: 0x5f595609, /127.0.0.1:39044 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.145 [DEBUG - New I/O worker #2311] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.145 [DEBUG - New I/O worker #2244] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.146 [DEBUG - New I/O worker #2253] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.146 [DEBUG - New I/O worker #2312] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.147 [DEBUG - New I/O worker #2312] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=794), Traces: [0ms] refreshing cache from master, [427ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [427ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [427ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [580ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [700ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [793ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.147 [DEBUG - New I/O worker #2312] (Deferred.java:1330) callback=retry RPC@1168005420 returned Deferred@1101120839(state=PENDING, result=null, callback=(continuation of Deferred@77812503 after retry RPC@1168005420), errback=(continuation of Deferred@77812503 after retry RPC@1168005420)), so the following Deferred is getting paused: Deferred@77812503(state=PAUSED, result=Deferred@1101120839, callback=org.apache.kudu.client.AsyncKuduClient$6@20d7da2 -> wakeup thread pool-3-thread-43, errback=passthrough -> wakeup thread pool-3-thread-43) 00:27:43.154 [DEBUG - New I/O worker #2295] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.154 [DEBUG - New I/O worker #2320] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.155 [DEBUG - New I/O worker #2320] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=681), Traces: [0ms] refreshing cache from master, [281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [282ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [465ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [556ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [681ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.155 [DEBUG - New I/O worker #2320] (Deferred.java:1330) callback=retry RPC@1346692643 returned Deferred@1417373939(state=PENDING, result=null, callback=(continuation of Deferred@293202968 after retry RPC@1346692643), errback=(continuation of Deferred@293202968 after retry RPC@1346692643)), so the following Deferred is getting paused: Deferred@293202968(state=PAUSED, result=Deferred@1417373939, callback=org.apache.kudu.client.AsyncKuduClient$6@57206aaf -> wakeup thread pool-3-thread-16, errback=passthrough -> wakeup thread pool-3-thread-16) 00:27:43.154 [DEBUG - New I/O worker #2292] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.156 [DEBUG - New I/O worker #2292] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.156 [DEBUG - New I/O worker #2292] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.156 [DEBUG - New I/O worker #2292] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.156 [DEBUG - New I/O worker #2292] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.157 [DEBUG - pool-3-thread-13] (Deferred.java:1330) callback=disconnect callback@473751972 returned Deferred@1235134396(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1720055292 after disconnect callback@473751972), errback=passthrough -> (continuation of Deferred@1720055292 after disconnect callback@473751972)), so the following Deferred is getting paused: Deferred@1720055292(state=PAUSED, result=Deferred@1235134396, callback=, errback=) 00:27:43.157 [DEBUG - New I/O worker #2299] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.158 [DEBUG - New I/O worker #2366] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.156 [DEBUG - New I/O worker #2313] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.159 [DEBUG - New I/O worker #2292] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.160 [DEBUG - New I/O worker #2298] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.161 [DEBUG - New I/O worker #2250] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.162 [DEBUG - New I/O worker #2278] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.164 [DEBUG - New I/O worker #2257] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.165 [DEBUG - New I/O worker #2300] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.166 [DEBUG - New I/O worker #2248] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.166 [DEBUG - New I/O worker #2286] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.169 [DEBUG - New I/O worker #2244] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.169 [DEBUG - New I/O worker #2244] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.169 [DEBUG - New I/O worker #2244] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.169 [DEBUG - New I/O worker #2244] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.169 [DEBUG - New I/O worker #2244] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.170 [DEBUG - pool-3-thread-11] (Deferred.java:1330) callback=disconnect callback@1203861947 returned Deferred@1689238794(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2102304499 after disconnect callback@1203861947), errback=passthrough -> (continuation of Deferred@2102304499 after disconnect callback@1203861947)), so the following Deferred is getting paused: Deferred@2102304499(state=PAUSED, result=Deferred@1689238794, callback=, errback=) 00:27:43.170 [DEBUG - New I/O worker #2270] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.171 [DEBUG - New I/O worker #2403] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.173 [DEBUG - New I/O worker #2244] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.174 [DEBUG - New I/O worker #2285] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.174 [DEBUG - New I/O worker #2285] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.174 [DEBUG - New I/O worker #2285] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.175 [DEBUG - New I/O worker #2285] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.175 [DEBUG - New I/O worker #2285] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.175 [DEBUG - pool-3-thread-28] (Deferred.java:1330) callback=disconnect callback@1374203143 returned Deferred@7088082(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1066284360 after disconnect callback@1374203143), errback=passthrough -> (continuation of Deferred@1066284360 after disconnect callback@1374203143)), so the following Deferred is getting paused: Deferred@1066284360(state=PAUSED, result=Deferred@7088082, callback=, errback=) 00:27:43.176 [DEBUG - New I/O worker #2537] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.176 [DEBUG - New I/O worker #2654] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.176 [DEBUG - New I/O worker #2275] (Negotiator.java:764) Authenticated connection [id: 0x568edba3, /127.0.0.1:39086 => /127.22.19.252:32833] using SASL/PLAIN 00:27:43.178 [DEBUG - New I/O worker #2307] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.178 [DEBUG - New I/O worker #2307] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=693), Traces: [0ms] refreshing cache from master, [141ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [141ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [141ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [189ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [189ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [693ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.178 [DEBUG - New I/O worker #2307] (Deferred.java:1330) callback=retry RPC@1765412036 returned Deferred@1829082427(state=PENDING, result=null, callback=(continuation of Deferred@671549183 after retry RPC@1765412036), errback=(continuation of Deferred@671549183 after retry RPC@1765412036)), so the following Deferred is getting paused: Deferred@671549183(state=PAUSED, result=Deferred@1829082427, callback=org.apache.kudu.client.AsyncKuduClient$6@2080ba12 -> wakeup thread pool-3-thread-61, errback=passthrough -> wakeup thread pool-3-thread-61) 00:27:43.179 [DEBUG - New I/O worker #2305] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.179 [DEBUG - New I/O worker #2312] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.179 [DEBUG - New I/O worker #2285] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.179 [DEBUG - New I/O worker #2305] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=693), Traces: [0ms] refreshing cache from master, [116ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [117ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [117ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [250ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [588ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [693ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.180 [DEBUG - New I/O worker #2305] (Deferred.java:1330) callback=retry RPC@1727301188 returned Deferred@1657474626(state=PENDING, result=null, callback=(continuation of Deferred@667520127 after retry RPC@1727301188), errback=(continuation of Deferred@667520127 after retry RPC@1727301188)), so the following Deferred is getting paused: Deferred@667520127(state=PAUSED, result=Deferred@1657474626, callback=org.apache.kudu.client.AsyncKuduClient$6@54a06741 -> wakeup thread pool-3-thread-68, errback=passthrough -> wakeup thread pool-3-thread-68) 00:27:43.180 [DEBUG - New I/O worker #2249] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.180 [DEBUG - New I/O worker #2249] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=828), Traces: [0ms] refreshing cache from master, [429ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [430ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [430ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [477ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [546ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [828ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.181 [DEBUG - New I/O worker #2249] (Deferred.java:1330) callback=retry RPC@1216733653 returned Deferred@1287289885(state=PENDING, result=null, callback=(continuation of Deferred@163113966 after retry RPC@1216733653), errback=(continuation of Deferred@163113966 after retry RPC@1216733653)), so the following Deferred is getting paused: Deferred@163113966(state=PAUSED, result=Deferred@1287289885, callback=org.apache.kudu.client.AsyncKuduClient$6@5f177ccb -> wakeup thread pool-3-thread-39, errback=passthrough -> wakeup thread pool-3-thread-39) 00:27:43.181 [DEBUG - New I/O worker #2292] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.181 [DEBUG - New I/O worker #2283] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.182 [DEBUG - New I/O worker #2283] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.182 [DEBUG - New I/O worker #2283] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.182 [DEBUG - New I/O worker #2283] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.182 [DEBUG - New I/O worker #2283] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.182 [DEBUG - New I/O worker #2246] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.183 [DEBUG - New I/O worker #2246] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.183 [DEBUG - New I/O worker #2246] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.183 [DEBUG - New I/O worker #2246] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.184 [DEBUG - New I/O worker #2246] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.184 [DEBUG - pool-3-thread-54] (Deferred.java:1330) callback=disconnect callback@1311481866 returned Deferred@254927418(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2066028700 after disconnect callback@1311481866), errback=passthrough -> (continuation of Deferred@2066028700 after disconnect callback@1311481866)), so the following Deferred is getting paused: Deferred@2066028700(state=PAUSED, result=Deferred@254927418, callback=, errback=) 00:27:43.185 [DEBUG - New I/O worker #2272] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.185 [DEBUG - New I/O worker #2536] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.186 [DEBUG - pool-3-thread-49] (Deferred.java:1330) callback=disconnect callback@282585274 returned Deferred@763313619(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1811365885 after disconnect callback@282585274), errback=passthrough -> (continuation of Deferred@1811365885 after disconnect callback@282585274)), so the following Deferred is getting paused: Deferred@1811365885(state=PAUSED, result=Deferred@763313619, callback=, errback=) 00:27:43.187 [DEBUG - New I/O worker #2488] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.187 [DEBUG - New I/O worker #2345] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.189 [DEBUG - New I/O worker #2246] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.189 [DEBUG - New I/O worker #2283] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.194 [DEBUG - New I/O worker #2246] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.195 [DEBUG - New I/O worker #2304] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.197 [DEBUG - New I/O worker #2320] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.197 [DEBUG - New I/O worker #2298] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.197 [DEBUG - New I/O worker #2298] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.198 [DEBUG - New I/O worker #2298] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.198 [DEBUG - New I/O worker #2298] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.198 [DEBUG - New I/O worker #2298] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.199 [DEBUG - New I/O worker #2306] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.199 [DEBUG - New I/O worker #2330] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.200 [DEBUG - pool-3-thread-35] (Deferred.java:1330) callback=disconnect callback@1685198615 returned Deferred@901728422(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@132252442 after disconnect callback@1685198615), errback=passthrough -> (continuation of Deferred@132252442 after disconnect callback@1685198615)), so the following Deferred is getting paused: Deferred@132252442(state=PAUSED, result=Deferred@901728422, callback=, errback=) 00:27:43.200 [DEBUG - New I/O worker #2547] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.201 [DEBUG - New I/O worker #2310] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.202 [DEBUG - New I/O worker #2298] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.202 [DEBUG - New I/O worker #2312] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.203 [DEBUG - New I/O worker #2312] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.203 [DEBUG - New I/O worker #2312] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.203 [DEBUG - New I/O worker #2312] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.203 [DEBUG - New I/O worker #2312] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.204 [DEBUG - pool-3-thread-43] (Deferred.java:1330) callback=disconnect callback@2137758814 returned Deferred@248970109(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1322594573 after disconnect callback@2137758814), errback=passthrough -> (continuation of Deferred@1322594573 after disconnect callback@2137758814)), so the following Deferred is getting paused: Deferred@1322594573(state=PAUSED, result=Deferred@248970109, callback=, errback=) 00:27:43.204 [DEBUG - New I/O worker #2343] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.204 [DEBUG - New I/O worker #2481] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.205 [DEBUG - New I/O worker #2285] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.205 [DEBUG - New I/O worker #2244] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.205 [DEBUG - New I/O worker #2249] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.206 [DEBUG - New I/O worker #2312] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.206 [DEBUG - New I/O worker #2334] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.207 [DEBUG - New I/O worker #2295] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.207 [DEBUG - New I/O worker #2295] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.208 [DEBUG - New I/O worker #2295] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.208 [DEBUG - New I/O worker #2295] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.208 [DEBUG - New I/O worker #2295] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.209 [DEBUG - New I/O worker #2298] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.209 [DEBUG - pool-3-thread-1] (Deferred.java:1330) callback=disconnect callback@482261233 returned Deferred@825644881(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1622459413 after disconnect callback@482261233), errback=passthrough -> (continuation of Deferred@1622459413 after disconnect callback@482261233)), so the following Deferred is getting paused: Deferred@1622459413(state=PAUSED, result=Deferred@825644881, callback=, errback=) 00:27:43.209 [DEBUG - New I/O worker #2332] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.209 [DEBUG - New I/O worker #2548] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.210 [DEBUG - New I/O worker #2295] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.212 [DEBUG - New I/O worker #2254] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.212 [DEBUG - New I/O worker #2295] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.213 [DEBUG - New I/O worker #2257] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.213 [DEBUG - New I/O worker #2257] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.213 [DEBUG - New I/O worker #2257] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.213 [DEBUG - New I/O worker #2257] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.214 [DEBUG - New I/O worker #2257] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.214 [DEBUG - Time-limited test] (Deferred.java:1330) callback=disconnect callback@322987424 returned Deferred@1501171651(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@236615984 after disconnect callback@322987424), errback=passthrough -> (continuation of Deferred@236615984 after disconnect callback@322987424)), so the following Deferred is getting paused: Deferred@236615984(state=PAUSED, result=Deferred@1501171651, callback=, errback=) 00:27:43.215 [DEBUG - New I/O worker #2437] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.215 [DEBUG - New I/O worker #2258] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.216 [DEBUG - New I/O worker #2257] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.222 [DEBUG - New I/O worker #2316] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.223 [DEBUG - New I/O worker #2316] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=1065), Traces: [0ms] refreshing cache from master, [119ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [119ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [119ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [381ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [410ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [1065ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.223 [DEBUG - New I/O worker #2257] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.223 [DEBUG - New I/O worker #2316] (Deferred.java:1330) callback=retry RPC@567451048 returned Deferred@636292426(state=PENDING, result=null, callback=(continuation of Deferred@1626314643 after retry RPC@567451048), errback=(continuation of Deferred@1626314643 after retry RPC@567451048)), so the following Deferred is getting paused: Deferred@1626314643(state=PAUSED, result=Deferred@636292426, callback=org.apache.kudu.client.AsyncKuduClient$6@644f1902 -> wakeup thread pool-3-thread-42, errback=passthrough -> wakeup thread pool-3-thread-42) 00:27:43.224 [DEBUG - New I/O worker #2271] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.225 [DEBUG - New I/O worker #2283] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.231 [DEBUG - New I/O worker #2312] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.232 [DEBUG - New I/O worker #2320] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.233 [DEBUG - New I/O worker #2304] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.233 [DEBUG - New I/O worker #2304] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.233 [DEBUG - New I/O worker #2304] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.233 [DEBUG - New I/O worker #2320] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.234 [DEBUG - New I/O worker #2320] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.234 [DEBUG - New I/O worker #2320] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.234 [DEBUG - New I/O worker #2320] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.236 [DEBUG - pool-3-thread-16] (Deferred.java:1330) callback=disconnect callback@1752479345 returned Deferred@98617531(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@898079069 after disconnect callback@1752479345), errback=passthrough -> (continuation of Deferred@898079069 after disconnect callback@1752479345)), so the following Deferred is getting paused: Deferred@898079069(state=PAUSED, result=Deferred@98617531, callback=, errback=) 00:27:43.236 [DEBUG - New I/O worker #2491] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.236 [DEBUG - New I/O worker #2318] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.237 [DEBUG - New I/O worker #2297] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.237 [DEBUG - New I/O worker #2297] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=893), Traces: [0ms] refreshing cache from master, [486ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [486ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [486ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [599ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [691ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [893ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.237 [DEBUG - New I/O worker #2297] (Deferred.java:1330) callback=retry RPC@1778743113 returned Deferred@1849355985(state=PENDING, result=null, callback=(continuation of Deferred@725122418 after retry RPC@1778743113), errback=(continuation of Deferred@725122418 after retry RPC@1778743113)), so the following Deferred is getting paused: Deferred@725122418(state=PAUSED, result=Deferred@1849355985, callback=org.apache.kudu.client.AsyncKuduClient$6@5ba70a39 -> wakeup thread pool-3-thread-17, errback=passthrough -> wakeup thread pool-3-thread-17) 00:27:43.238 [DEBUG - New I/O worker #2582] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.239 [DEBUG - New I/O worker #2263] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.234 [DEBUG - New I/O worker #2287] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.237 [DEBUG - New I/O worker #2306] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.240 [DEBUG - New I/O worker #2249] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.240 [DEBUG - New I/O worker #2249] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.240 [DEBUG - New I/O worker #2249] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.240 [DEBUG - New I/O worker #2320] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.240 [DEBUG - New I/O worker #2306] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.241 [DEBUG - New I/O worker #2306] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.241 [DEBUG - New I/O worker #2306] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.241 [DEBUG - New I/O worker #2306] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.242 [DEBUG - pool-3-thread-51] (Deferred.java:1330) callback=disconnect callback@1902798799 returned Deferred@1613734191(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1685636535 after disconnect callback@1902798799), errback=passthrough -> (continuation of Deferred@1685636535 after disconnect callback@1902798799)), so the following Deferred is getting paused: Deferred@1685636535(state=PAUSED, result=Deferred@1613734191, callback=, errback=) 00:27:43.242 [DEBUG - New I/O worker #2342] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.243 [DEBUG - New I/O worker #2487] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.241 [DEBUG - New I/O worker #2249] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.244 [DEBUG - New I/O worker #2249] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.245 [DEBUG - pool-3-thread-39] (Deferred.java:1330) callback=disconnect callback@438916686 returned Deferred@1264493817(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@2143757201 after disconnect callback@438916686), errback=passthrough -> (continuation of Deferred@2143757201 after disconnect callback@438916686)), so the following Deferred is getting paused: Deferred@2143757201(state=PAUSED, result=Deferred@1264493817, callback=, errback=) 00:27:43.245 [DEBUG - New I/O worker #2425] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.240 [DEBUG - New I/O worker #2304] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.246 [DEBUG - New I/O worker #2304] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.247 [DEBUG - pool-3-thread-56] (Deferred.java:1330) callback=disconnect callback@799087534 returned Deferred@2020894204(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1009105383 after disconnect callback@799087534), errback=passthrough -> (continuation of Deferred@1009105383 after disconnect callback@799087534)), so the following Deferred is getting paused: Deferred@1009105383(state=PAUSED, result=Deferred@2020894204, callback=, errback=) 00:27:43.247 [DEBUG - New I/O worker #2349] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.245 [DEBUG - New I/O worker #2266] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.247 [DEBUG - New I/O worker #2289] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.247 [DEBUG - New I/O worker #2482] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.248 [DEBUG - New I/O worker #2265] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.250 [DEBUG - New I/O worker #2249] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.250 [DEBUG - New I/O worker #2306] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.251 [DEBUG - New I/O worker #2304] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.252 [DEBUG - New I/O worker #2249] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.252 [DEBUG - New I/O worker #2254] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.253 [DEBUG - New I/O worker #2254] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.253 [DEBUG - New I/O worker #2254] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.253 [DEBUG - New I/O worker #2254] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.253 [DEBUG - New I/O worker #2320] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.253 [DEBUG - New I/O worker #2254] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.254 [DEBUG - pool-3-thread-31] (Deferred.java:1330) callback=disconnect callback@788977254 returned Deferred@119936865(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@395867850 after disconnect callback@788977254), errback=passthrough -> (continuation of Deferred@395867850 after disconnect callback@788977254)), so the following Deferred is getting paused: Deferred@395867850(state=PAUSED, result=Deferred@119936865, callback=, errback=) 00:27:43.255 [DEBUG - New I/O worker #2261] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.255 [DEBUG - New I/O worker #2336] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.256 [DEBUG - New I/O worker #2297] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.257 [DEBUG - New I/O worker #2254] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.258 [DEBUG - New I/O worker #2254] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.259 [DEBUG - New I/O worker #2306] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.260 [DEBUG - New I/O worker #2304] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.270 [DEBUG - New I/O worker #2310] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.271 [DEBUG - New I/O worker #2263] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.271 [DEBUG - New I/O worker #2263] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.271 [DEBUG - New I/O worker #2263] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.271 [DEBUG - New I/O worker #2263] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.271 [DEBUG - New I/O worker #2310] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.272 [DEBUG - New I/O worker #2310] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.272 [DEBUG - New I/O worker #2310] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.273 [DEBUG - New I/O worker #2310] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.275 [DEBUG - New I/O worker #2297] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.275 [DEBUG - New I/O worker #2297] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.276 [DEBUG - New I/O worker #2297] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.276 [DEBUG - New I/O worker #2297] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.276 [DEBUG - New I/O worker #2297] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.276 [DEBUG - New I/O worker #2250] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.277 [DEBUG - New I/O worker #2250] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.277 [DEBUG - New I/O worker #2250] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.277 [DEBUG - New I/O worker #2250] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.277 [DEBUG - New I/O worker #2250] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.278 [DEBUG - pool-3-thread-17] (Deferred.java:1330) callback=disconnect callback@1840706297 returned Deferred@565654576(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1395670460 after disconnect callback@1840706297), errback=passthrough -> (continuation of Deferred@1395670460 after disconnect callback@1840706297)), so the following Deferred is getting paused: Deferred@1395670460(state=PAUSED, result=Deferred@565654576, callback=, errback=) 00:27:43.278 [DEBUG - pool-3-thread-37] (Deferred.java:1330) callback=disconnect callback@874890999 returned Deferred@1710167297(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1656464920 after disconnect callback@874890999), errback=passthrough -> (continuation of Deferred@1656464920 after disconnect callback@874890999)), so the following Deferred is getting paused: Deferred@1656464920(state=PAUSED, result=Deferred@1710167297, callback=, errback=) 00:27:43.278 [DEBUG - New I/O worker #2329] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.279 [DEBUG - New I/O worker #2521] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.279 [DEBUG - New I/O worker #2279] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.279 [DEBUG - New I/O worker #2420] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.280 [DEBUG - New I/O worker #2303] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.283 [DEBUG - pool-3-thread-71] (Deferred.java:1330) callback=disconnect callback@515927836 returned Deferred@876929340(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1694808234 after disconnect callback@515927836), errback=passthrough -> (continuation of Deferred@1694808234 after disconnect callback@515927836)), so the following Deferred is getting paused: Deferred@1694808234(state=PAUSED, result=Deferred@876929340, callback=, errback=) 00:27:43.283 [DEBUG - New I/O worker #2492] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.284 [DEBUG - New I/O worker #2493] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.271 [DEBUG - New I/O worker #2321] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.284 [DEBUG - New I/O worker #2321] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.284 [DEBUG - New I/O worker #2321] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.284 [DEBUG - New I/O worker #2321] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.285 [DEBUG - New I/O worker #2321] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.271 [DEBUG - New I/O worker #2318] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.285 [DEBUG - New I/O worker #2318] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.286 [DEBUG - New I/O worker #2318] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.286 [DEBUG - New I/O worker #2318] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.286 [DEBUG - New I/O worker #2318] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.287 [DEBUG - pool-3-thread-77] (Deferred.java:1330) callback=disconnect callback@834806275 returned Deferred@1525640596(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1022489608 after disconnect callback@834806275), errback=passthrough -> (continuation of Deferred@1022489608 after disconnect callback@834806275)), so the following Deferred is getting paused: Deferred@1022489608(state=PAUSED, result=Deferred@1525640596, callback=, errback=) 00:27:43.287 [DEBUG - pool-3-thread-7] (Deferred.java:1330) callback=disconnect callback@907599985 returned Deferred@763546898(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@278259760 after disconnect callback@907599985), errback=passthrough -> (continuation of Deferred@278259760 after disconnect callback@907599985)), so the following Deferred is getting paused: Deferred@278259760(state=PAUSED, result=Deferred@763546898, callback=, errback=) 00:27:43.287 [DEBUG - New I/O worker #2351] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.288 [DEBUG - New I/O worker #2523] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.288 [DEBUG - New I/O worker #2525] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.289 [DEBUG - New I/O worker #2250] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.289 [DEBUG - New I/O worker #2287] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.289 [DEBUG - New I/O worker #2287] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.289 [DEBUG - New I/O worker #2263] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299982, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299982, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299982, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299982, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.290 [DEBUG - pool-3-thread-20] (Deferred.java:1330) callback=disconnect callback@191874100 returned Deferred@911526274(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1697629716 after disconnect callback@191874100), errback=passthrough -> (continuation of Deferred@1697629716 after disconnect callback@191874100)), so the following Deferred is getting paused: Deferred@1697629716(state=PAUSED, result=Deferred@911526274, callback=, errback=) 00:27:43.290 [DEBUG - New I/O worker #2293] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.290 [DEBUG - New I/O worker #2307] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.276 [DEBUG - New I/O worker #2308] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.291 [DEBUG - New I/O worker #2308] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=807), Traces: [0ms] refreshing cache from master, [168ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [169ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [169ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [264ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [322ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [792ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.291 [DEBUG - New I/O worker #2316] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.292 [DEBUG - New I/O worker #2297] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.293 [DEBUG - New I/O worker #2263] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.276 [DEBUG - New I/O worker #2281] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.293 [DEBUG - New I/O worker #2281] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=962), Traces: [0ms] refreshing cache from master, [500ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [500ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [500ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [707ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [745ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [945ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.293 [DEBUG - New I/O worker #2281] (Deferred.java:1330) callback=retry RPC@352056780 returned Deferred@281305995(state=PENDING, result=null, callback=(continuation of Deferred@1439105015 after retry RPC@352056780), errback=(continuation of Deferred@1439105015 after retry RPC@352056780)), so the following Deferred is getting paused: Deferred@1439105015(state=PAUSED, result=Deferred@281305995, callback=org.apache.kudu.client.AsyncKuduClient$6@2f474e64 -> wakeup thread pool-3-thread-65, errback=passthrough -> wakeup thread pool-3-thread-65) 00:27:43.293 [DEBUG - New I/O worker #2278] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.294 [DEBUG - New I/O worker #2278] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.294 [DEBUG - New I/O worker #2278] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.294 [DEBUG - New I/O worker #2278] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.294 [DEBUG - New I/O worker #2278] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.295 [DEBUG - pool-3-thread-22] (Deferred.java:1330) callback=disconnect callback@1690582013 returned Deferred@494422828(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1369199216 after disconnect callback@1690582013), errback=passthrough -> (continuation of Deferred@1369199216 after disconnect callback@1690582013)), so the following Deferred is getting paused: Deferred@1369199216(state=PAUSED, result=Deferred@494422828, callback=, errback=) 00:27:43.295 [DEBUG - New I/O worker #2461] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.295 [DEBUG - New I/O worker #2393] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.292 [DEBUG - New I/O worker #2252] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.296 [DEBUG - New I/O worker #2252] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=819), Traces: [0ms] refreshing cache from master, [227ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [227ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [228ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [278ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [402ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [816ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.296 [DEBUG - New I/O worker #2252] (Deferred.java:1330) callback=retry RPC@47891141 returned Deferred@115693126(state=PENDING, result=null, callback=(continuation of Deferred@1139263742 after retry RPC@47891141), errback=(continuation of Deferred@1139263742 after retry RPC@47891141)), so the following Deferred is getting paused: Deferred@1139263742(state=PAUSED, result=Deferred@115693126, callback=org.apache.kudu.client.AsyncKuduClient$6@26f6a939 -> wakeup thread pool-3-thread-8, errback=passthrough -> wakeup thread pool-3-thread-8) 00:27:43.292 [DEBUG - New I/O worker #2247] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.297 [DEBUG - New I/O worker #2247] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=819), Traces: [0ms] refreshing cache from master, [275ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [275ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [275ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [576ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [592ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [815ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.297 [DEBUG - New I/O worker #2247] (Deferred.java:1330) callback=retry RPC@2094600506 returned Deferred@2028378745(state=PENDING, result=null, callback=(continuation of Deferred@1038359297 after retry RPC@2094600506), errback=(continuation of Deferred@1038359297 after retry RPC@2094600506)), so the following Deferred is getting paused: Deferred@1038359297(state=PAUSED, result=Deferred@2028378745, callback=org.apache.kudu.client.AsyncKuduClient$6@3877330d -> wakeup thread pool-3-thread-72, errback=passthrough -> wakeup thread pool-3-thread-72) 00:27:43.291 [DEBUG - New I/O worker #2308] (Deferred.java:1330) callback=retry RPC@2023336055 returned Deferred@2091262908(state=PENDING, result=null, callback=(continuation of Deferred@967090764 after retry RPC@2023336055), errback=(continuation of Deferred@967090764 after retry RPC@2023336055)), so the following Deferred is getting paused: Deferred@967090764(state=PAUSED, result=Deferred@2091262908, callback=org.apache.kudu.client.AsyncKuduClient$6@3db8959c -> wakeup thread pool-3-thread-45, errback=passthrough -> wakeup thread pool-3-thread-45) 00:27:43.291 [DEBUG - New I/O worker #2318] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.290 [DEBUG - New I/O worker #2340] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.289 [DEBUG - New I/O worker #2287] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.298 [DEBUG - New I/O worker #2287] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.298 [DEBUG - New I/O worker #2287] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.287 [DEBUG - New I/O worker #2485] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.299 [DEBUG - New I/O worker #2278] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.299 [DEBUG - New I/O worker #2483] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.299 [DEBUG - New I/O worker #2265] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.300 [DEBUG - New I/O worker #2265] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.300 [DEBUG - New I/O worker #2265] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.301 [DEBUG - New I/O worker #2265] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.301 [DEBUG - New I/O worker #2265] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.302 [DEBUG - pool-3-thread-30] (Deferred.java:1330) callback=disconnect callback@625189694 returned Deferred@177463562(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@51125714 after disconnect callback@625189694), errback=passthrough -> (continuation of Deferred@51125714 after disconnect callback@625189694)), so the following Deferred is getting paused: Deferred@51125714(state=PAUSED, result=Deferred@177463562, callback=, errback=) 00:27:43.302 [DEBUG - New I/O worker #2335] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.302 [DEBUG - New I/O worker #2528] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.299 [DEBUG - New I/O worker #2350] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.299 [DEBUG - pool-3-thread-58] (Deferred.java:1330) callback=disconnect callback@1377697889 returned Deferred@379453949(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@951219220 after disconnect callback@1377697889), errback=passthrough -> (continuation of Deferred@951219220 after disconnect callback@1377697889)), so the following Deferred is getting paused: Deferred@951219220(state=PAUSED, result=Deferred@379453949, callback=, errback=) 00:27:43.302 [DEBUG - New I/O worker #2271] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.303 [DEBUG - New I/O worker #2271] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.303 [DEBUG - New I/O worker #2271] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.303 [DEBUG - New I/O worker #2271] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.303 [DEBUG - New I/O worker #2271] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.300 [DEBUG - New I/O worker #2287] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.304 [DEBUG - pool-3-thread-47] (Deferred.java:1330) callback=disconnect callback@798191495 returned Deferred@2037490922(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@416559559 after disconnect callback@798191495), errback=passthrough -> (continuation of Deferred@416559559 after disconnect callback@798191495)), so the following Deferred is getting paused: Deferred@416559559(state=PAUSED, result=Deferred@2037490922, callback=, errback=) 00:27:43.304 [DEBUG - New I/O worker #2404] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.304 [DEBUG - New I/O worker #2275] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:43.305 [DEBUG - New I/O worker #2275] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=975), Traces: [0ms] refreshing cache from master, [505ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.252:32833, [505ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.254:44849, [505ms] Sub RPC ConnectToMaster: sending RPC to server master-127.22.19.253:37879, [671ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.253:37879: OK, [771ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.254:44849: OK, [974ms] Sub RPC ConnectToMaster: received response from server master-127.22.19.252:32833: OK, deferred=null) after lookup 00:27:43.305 [DEBUG - New I/O worker #2275] (Deferred.java:1330) callback=retry RPC@2067674115 returned Deferred@2130821052(state=PENDING, result=null, callback=(continuation of Deferred@973291064 after retry RPC@2067674115), errback=(continuation of Deferred@973291064 after retry RPC@2067674115)), so the following Deferred is getting paused: Deferred@973291064(state=PAUSED, result=Deferred@2130821052, callback=org.apache.kudu.client.AsyncKuduClient$6@167d200 -> wakeup thread pool-3-thread-40, errback=passthrough -> wakeup thread pool-3-thread-40) 00:27:43.306 [DEBUG - New I/O worker #2305] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.304 [DEBUG - New I/O worker #2402] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.307 [DEBUG - New I/O worker #2271] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.307 [DEBUG - New I/O worker #2265] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.308 [DEBUG - New I/O worker #2321] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.311 [DEBUG - New I/O worker #2310] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.318 [DEBUG - New I/O worker #2265] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.319 [DEBUG - New I/O worker #2297] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.319 [DEBUG - New I/O worker #2278] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.320 [DEBUG - New I/O worker #2289] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.320 [DEBUG - New I/O worker #2289] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.320 [DEBUG - New I/O worker #2289] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.320 [DEBUG - New I/O worker #2289] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.320 [DEBUG - New I/O worker #2289] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.321 [DEBUG - pool-3-thread-48] (Deferred.java:1330) callback=disconnect callback@1412164294 returned Deferred@1533308996(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1751598795 after disconnect callback@1412164294), errback=passthrough -> (continuation of Deferred@1751598795 after disconnect callback@1412164294)), so the following Deferred is getting paused: Deferred@1751598795(state=PAUSED, result=Deferred@1533308996, callback=, errback=) 00:27:43.321 [DEBUG - New I/O worker #2416] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.322 [DEBUG - New I/O worker #2451] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.322 [DEBUG - New I/O worker #2289] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.323 [DEBUG - New I/O worker #2318] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.323 [DEBUG - New I/O worker #2250] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.324 [DEBUG - New I/O worker #2287] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.324 [DEBUG - New I/O worker #2263] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.324 [DEBUG - New I/O worker #2271] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.327 [DEBUG - New I/O worker #2321] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.327 [DEBUG - New I/O worker #2310] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.339 [DEBUG - New I/O worker #2289] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.339 [DEBUG - New I/O worker #2307] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.340 [DEBUG - New I/O worker #2307] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.340 [DEBUG - New I/O worker #2307] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.340 [DEBUG - New I/O worker #2307] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.340 [DEBUG - New I/O worker #2307] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.341 [DEBUG - pool-3-thread-61] (Deferred.java:1330) callback=disconnect callback@537583982 returned Deferred@511607683(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1548912581 after disconnect callback@537583982), errback=passthrough -> (continuation of Deferred@1548912581 after disconnect callback@537583982)), so the following Deferred is getting paused: Deferred@1548912581(state=PAUSED, result=Deferred@511607683, callback=, errback=) 00:27:43.341 [DEBUG - New I/O worker #2346] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.342 [DEBUG - New I/O worker #2586] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.343 [DEBUG - New I/O worker #2307] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.344 [DEBUG - New I/O worker #2308] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.344 [DEBUG - New I/O worker #2303] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.344 [DEBUG - New I/O worker #2303] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.344 [DEBUG - New I/O worker #2303] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.345 [DEBUG - New I/O worker #2303] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.345 [DEBUG - New I/O worker #2303] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.345 [DEBUG - pool-3-thread-59] (Deferred.java:1330) callback=disconnect callback@789399039 returned Deferred@1047921920(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1330204240 after disconnect callback@789399039), errback=passthrough -> (continuation of Deferred@1330204240 after disconnect callback@789399039)), so the following Deferred is getting paused: Deferred@1330204240(state=PAUSED, result=Deferred@1047921920, callback=, errback=) 00:27:43.346 [DEBUG - New I/O worker #2496] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.346 [DEBUG - New I/O worker #2495] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.344 [DEBUG - New I/O worker #2281] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.346 [DEBUG - New I/O worker #2303] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.347 [DEBUG - New I/O worker #2307] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.351 [DEBUG - New I/O worker #2303] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.352 [DEBUG - New I/O worker #2252] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.352 [DEBUG - New I/O worker #2247] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.352 [DEBUG - New I/O worker #2316] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.353 [DEBUG - New I/O worker #2316] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.353 [DEBUG - New I/O worker #2316] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.353 [DEBUG - New I/O worker #2316] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.354 [DEBUG - New I/O worker #2316] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.354 [DEBUG - pool-3-thread-42] (Deferred.java:1330) callback=disconnect callback@826409199 returned Deferred@712728604(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@208690706 after disconnect callback@826409199), errback=passthrough -> (continuation of Deferred@208690706 after disconnect callback@826409199)), so the following Deferred is getting paused: Deferred@208690706(state=PAUSED, result=Deferred@712728604, callback=, errback=) 00:27:43.355 [DEBUG - New I/O worker #2326] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.355 [DEBUG - New I/O worker #2508] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.356 [DEBUG - New I/O worker #2316] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.363 [DEBUG - New I/O worker #2305] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.363 [DEBUG - New I/O worker #2275] (AsyncKuduClient.java:761) Opened table f846a8dbe51f43e08571b061f393845f 00:27:43.363 [DEBUG - New I/O worker #2316] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.363 [DEBUG - New I/O worker #2305] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.364 [DEBUG - New I/O worker #2305] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.364 [DEBUG - New I/O worker #2305] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.365 [DEBUG - New I/O worker #2305] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.371 [DEBUG - New I/O worker #2305] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.371 [DEBUG - pool-3-thread-68] (Deferred.java:1330) callback=disconnect callback@391882028 returned Deferred@269238861(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1016254311 after disconnect callback@391882028), errback=passthrough -> (continuation of Deferred@1016254311 after disconnect callback@391882028)), so the following Deferred is getting paused: Deferred@1016254311(state=PAUSED, result=Deferred@269238861, callback=, errback=) 00:27:43.372 [DEBUG - New I/O worker #2281] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.372 [DEBUG - New I/O worker #2281] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.372 [DEBUG - New I/O worker #2308] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.373 [DEBUG - New I/O worker #2308] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.373 [DEBUG - New I/O worker #2308] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.373 [DEBUG - New I/O worker #2308] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.373 [DEBUG - New I/O worker #2308] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.374 [DEBUG - pool-3-thread-45] (Deferred.java:1330) callback=disconnect callback@1823021606 returned Deferred@552206317(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@725161716 after disconnect callback@1823021606), errback=passthrough -> (continuation of Deferred@725161716 after disconnect callback@1823021606)), so the following Deferred is getting paused: Deferred@725161716(state=PAUSED, result=Deferred@552206317, callback=, errback=) 00:27:43.374 [DEBUG - New I/O worker #2647] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.372 [DEBUG - New I/O worker #2281] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.374 [DEBUG - New I/O worker #2281] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.374 [DEBUG - New I/O worker #2281] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.374 [DEBUG - New I/O worker #2490] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.374 [DEBUG - New I/O worker #2347] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.373 [DEBUG - New I/O worker #2484] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.375 [DEBUG - New I/O worker #2506] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.376 [DEBUG - New I/O worker #2247] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.376 [DEBUG - New I/O worker #2247] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.376 [DEBUG - New I/O worker #2247] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.376 [DEBUG - New I/O worker #2247] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.376 [DEBUG - New I/O worker #2247] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.375 [DEBUG - New I/O worker #2337] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.377 [DEBUG - pool-3-thread-72] (Deferred.java:1330) callback=disconnect callback@1857063536 returned Deferred@217793447(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@73892269 after disconnect callback@1857063536), errback=passthrough -> (continuation of Deferred@73892269 after disconnect callback@1857063536)), so the following Deferred is getting paused: Deferred@73892269(state=PAUSED, result=Deferred@217793447, callback=, errback=) 00:27:43.377 [DEBUG - New I/O worker #2252] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.377 [DEBUG - New I/O worker #2252] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.377 [DEBUG - New I/O worker #2252] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.378 [DEBUG - New I/O worker #2252] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.375 [DEBUG - pool-3-thread-65] (Deferred.java:1330) callback=disconnect callback@118808051 returned Deferred@312549211(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@158529136 after disconnect callback@118808051), errback=passthrough -> (continuation of Deferred@158529136 after disconnect callback@118808051)), so the following Deferred is getting paused: Deferred@158529136(state=PAUSED, result=Deferred@312549211, callback=, errback=) 00:27:43.377 [DEBUG - New I/O worker #2534] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.377 [DEBUG - New I/O worker #2323] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.376 [DEBUG - New I/O worker #2275] (AsyncKuduClient.java:2118) Learned about tablet e3df67dd290b4bf6b4b9bc367348c886 for table 'testManyClients' with partition [, 0x8000000A) 00:27:43.379 [DEBUG - New I/O worker #2275] (AsyncKuduClient.java:2118) Learned about tablet fa4fe6266d3b4911aaac3bcebe15999b for table 'testManyClients' with partition [0x8000000A, 0x80000014) 00:27:43.379 [DEBUG - New I/O worker #2275] (AsyncKuduClient.java:2118) Learned about tablet d5cb2974aabf40dea3f204d4d5929375 for table 'testManyClients' with partition [0x80000014, 0x8000001E) 00:27:43.379 [DEBUG - New I/O worker #2275] (AsyncKuduClient.java:2118) Learned about tablet 762f4bc1622449b8868b95d67f43e131 for table 'testManyClients' with partition [0x8000001E, ) 00:27:43.379 [DEBUG - New I/O worker #2275] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299999, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299999, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.380 [DEBUG - New I/O worker #2275] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.380 [DEBUG - New I/O worker #2338] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.380 [DEBUG - New I/O worker #2510] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.378 [DEBUG - New I/O worker #2247] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.380 [DEBUG - pool-3-thread-40] (Deferred.java:1330) callback=disconnect callback@1521804371 returned Deferred@1428372271(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1205257301 after disconnect callback@1521804371), errback=passthrough -> (continuation of Deferred@1205257301 after disconnect callback@1521804371)), so the following Deferred is getting paused: Deferred@1205257301(state=PAUSED, result=Deferred@1428372271, callback=, errback=) 00:27:43.379 [DEBUG - New I/O worker #2252] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x8000000A, ttl=299999, tablet=e3df67dd290b4bf6b4b9bc367348c886@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}, Tablet{lowerBoundPartitionKey=0x8000000A, upperBoundPartitionKey=0x80000014, ttl=299999, tablet=fa4fe6266d3b4911aaac3bcebe15999b@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x80000014, upperBoundPartitionKey=0x8000001E, ttl=299998, tablet=d5cb2974aabf40dea3f204d4d5929375@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751),5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)[L]]}, Tablet{lowerBoundPartitionKey=0x8000001E, upperBoundPartitionKey=0x, ttl=299998, tablet=762f4bc1622449b8868b95d67f43e131@[1313c80b0ed640658101132ab87b8fc5(127.22.19.194:43751)[L],5ab94ed628b14cf09ec9bd11be159caa(127.22.19.193:35627),c9ad861f849642ad801dde9f5e92c23f(127.22.19.195:44599)]}] 00:27:43.382 [DEBUG - pool-3-thread-8] (Deferred.java:1330) callback=disconnect callback@744437263 returned Deferred@2014534963(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1997893415 after disconnect callback@744437263), errback=passthrough -> (continuation of Deferred@1997893415 after disconnect callback@744437263)), so the following Deferred is getting paused: Deferred@1997893415(state=PAUSED, result=Deferred@2014534963, callback=, errback=) 00:27:43.382 [DEBUG - New I/O worker #2268] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.382 [DEBUG - New I/O worker #2494] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.379 [DEBUG - New I/O worker #2308] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.379 [DEBUG - New I/O worker #2281] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.385 [DEBUG - New I/O worker #2252] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.391 [DEBUG - New I/O worker #2510] (Connection.java:688) [peer master-127.22.19.253:37879(127.22.19.253:37879)] cleaning up while in state READY due to: connection disconnected 00:27:43.395 [DEBUG - New I/O worker #2247] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.398 [DEBUG - New I/O worker #2281] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.399 [DEBUG - New I/O worker #2308] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.400 [DEBUG - New I/O worker #2347] (Connection.java:688) [peer master-127.22.19.254:44849(127.22.19.254:44849)] cleaning up while in state READY due to: connection disconnected 00:27:43.403 [DEBUG - New I/O worker #2252] (Connection.java:688) [peer master-127.22.19.252:32833(127.22.19.252:32833)] cleaning up while in state READY due to: connection disconnected 00:27:43.409 [DEBUG - main] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:27:43.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:43.420084 22607 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver with pid 22808 00:27:43.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.446674 23114 connection.cc:647] server connection from 127.22.19.193:50145 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.193:50145): Connection reset by peer (error 104) 00:27:43.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.446655 22985 connection.cc:647] server connection from 127.22.19.193:47619 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.193:47619): Connection reset by peer (error 104) 00:27:43.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.447701 23116 connection.cc:647] client connection to 127.22.19.193:35627 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.193:35627): Connection reset by peer (error 104) 00:27:43.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.448624 22985 connection.cc:647] client connection to 127.22.19.193:35627 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.193:35627): Connection reset by peer (error 104) 00:27:43.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.448650 22624 connection.cc:647] server connection from 127.22.19.193:50263 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.193:50263): Connection reset by peer (error 104) 00:27:43.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.449460 22689 connection.cc:647] server connection from 127.22.19.193:40199 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.193:40199): Connection reset by peer (error 104) 00:27:43.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.450395 22758 connection.cc:647] client connection to 127.22.19.193:35627 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.193:35627): Connection reset by peer (error 104) 00:27:43.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.450395 22755 connection.cc:647] server connection from 127.22.19.193:47609 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.193:47609): Connection reset by peer (error 104) 00:27:43.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:43.452564 22607 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver with pid 22969 00:27:43.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.465008 23116 consensus_peers.cc:457] T fa4fe6266d3b4911aaac3bcebe15999b P c9ad861f849642ad801dde9f5e92c23f -> Peer 5ab94ed628b14cf09ec9bd11be159caa (127.22.19.193:35627): Couldn't send request to peer 5ab94ed628b14cf09ec9bd11be159caa. Status: Network error: Client connection negotiation failed: client connection to 127.22.19.193:35627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 00:27:43.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.490250 22688 connection.cc:647] server connection from 127.22.19.194:37075 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.194:37075): Connection reset by peer (error 104) 00:27:43.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.490254 23116 connection.cc:647] client connection to 127.22.19.194:43751 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.194:43751): Connection reset by peer (error 104) 00:27:43.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.490262 22625 connection.cc:647] server connection from 127.22.19.194:51479 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.194:51479): Connection reset by peer (error 104) 00:27:43.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.490312 22754 connection.cc:647] server connection from 127.22.19.194:37147 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.194:37147): Connection reset by peer (error 104) 00:27:43.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.491014 23116 connection.cc:647] server connection from 127.22.19.194:56647 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.194:56647): Connection reset by peer (error 104) 00:27:43.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.491374 22758 connection.cc:647] client connection to 127.22.19.194:43751 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.194:43751): Connection reset by peer (error 104) 00:27:43.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:43.491828 22607 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver with pid 23098 00:27:43.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.512534 22626 connection.cc:647] server connection from 127.22.19.195:49809 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.195:49809): Connection reset by peer (error 104) 00:27:43.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.512545 22692 connection.cc:647] server connection from 127.22.19.195:48051 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.195:48051): Connection reset by peer (error 104) 00:27:43.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.512645 22754 connection.cc:647] server connection from 127.22.19.195:55151 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.195:55151): Connection reset by peer (error 104) 00:27:43.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.513283 22758 connection.cc:647] client connection to 127.22.19.195:44599 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.195:44599): Connection reset by peer (error 104) 00:27:43.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:43.513756 22607 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master with pid 22616 00:27:43.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.538221 22689 connection.cc:647] server connection from 127.0.0.1:39154 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39154): Connection reset by peer (error 104) 00:27:43.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.538331 22758 connection.cc:647] server connection from 127.0.0.1:37200 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37200): Connection reset by peer (error 104) 00:27:43.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.539000 22688 connection.cc:647] client connection to 127.22.19.254:44849 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.254:44849): Connection reset by peer (error 104) 00:27:43.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:43.539681 22607 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master with pid 22678 00:27:43.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.539782 22753 connection.cc:647] client connection to 127.22.19.254:44849 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) 00:27:43.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.541095 22753 consensus_peers.cc:457] T 00000000000000000000000000000000 P 02a20b80b4f7488e9b6969d470490d29 -> Peer 3a6002cd74bd48eca090ad5ae522ab7d (127.22.19.254:44849): Couldn't send request to peer 3a6002cd74bd48eca090ad5ae522ab7d. Status: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry. 00:27:43.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.544137 22753 connection.cc:320] Error closing socket: Network error: TlsSocket::Close: Broken pipe 00:27:43.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.569677 22758 connection.cc:647] client connection to 127.22.19.253:37879 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.253:37879): Connection reset by peer (error 104) 00:27:43.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.570462 22754 connection.cc:647] server connection from 127.0.0.1:37202 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37202): Connection reset by peer (error 104) 00:27:43.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:43.572177 22607 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master with pid 22742 00:27:43.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:43.538421 22615 connection.cc:647] client connection to 127.22.19.254:44849 recv error: Network error: failed to read from TLS socket (remote: 127.22.19.254:44849): Connection reset by peer (error 104) 00:27:44.989 [INFO - main] (RandomUtils.java:46) Using random seed: 1550622464989 .00:27:44.989 [INFO - main] (KuduTestHarness.java:140) Creating a new MiniKuduCluster... 00:27:44.989 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskc0vaMl/test-tmp 00:27:44.989 [INFO - main] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin 00:27:44.990 [INFO - main] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskc0vaMl/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb] 00:27:44.991 [DEBUG - main] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 3 enable_kerberos: false cluster_root: "/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632" hms_mode: NONE mini_kdc_options { } } 00:27:45.154 [DEBUG - main] (MiniKuduCluster.java:165) Response: 00:27:45.154 [DEBUG - main] (MiniKuduCluster.java:155) Request: start_cluster { } 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.177867 27163 external_mini_cluster.cc:933] Running /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/wal 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/data 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.26.134.254:39417 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.26.134.254 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/logs 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/data/info.pb 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 00:27:45.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.26.134.254:39417,127.26.134.253:40601,127.26.134.252:33715 with env {} 00:27:45.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 00:27:45.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:45.314482 27171 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 00:27:45.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:45.314929 27171 flags.cc:406] Enabled unsafe flag: --never_fsync=true 00:27:45.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:45.326578 27171 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 00:27:45.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:45.326756 27171 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 00:27:45.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:45.326923 27171 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 00:27:45.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:45.327122 27171 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.344060 27171 master_main.cc:73] Master server non-default flags: 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/data 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/wal 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.26.134.254:39417,127.26.134.253:40601,127.26.134.252:33715 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.26.134.254:39417 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/data/info.pb 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.26.134.254 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/logs 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-SNAPSHOT 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5673617937533e44afc5412722108dfed4a7f7b2 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by None at 20 Feb 2019 00:18:11 UTC on jenkins-slave-8qw8 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 16477 00:27:45.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 00:27:45.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.358141 27171 master_main.cc:80] Initializing master server... 00:27:45.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.359803 27171 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 94671us 00:27:45.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.360780 27171 fs_manager.cc:263] Metadata directory not provided 00:27:45.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.361094 27171 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 00:27:45.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.361439 27171 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 00:27:45.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.361562 27171 server_base.cc:434] Attempting to create new FS layout instead 00:27:45.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.364603 27171 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/data/instance: 00:27:45.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "11bb274b603a4ca09d30337d047b71c5" 00:27:45.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:45 on dist-test-slave-dist-test-slave-vnz7" 00:27:45.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.365317 27171 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/wal/instance: 00:27:45.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "11bb274b603a4ca09d30337d047b71c5" 00:27:45.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:45 on dist-test-slave-dist-test-slave-vnz7" 00:27:45.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.376085 27171 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.001s 00:27:45.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.377562 27171 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 00:27:45.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.378094 27171 file_cache.cc:466] Constructed file cache lbm with capacity 419430 00:27:45.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.385236 27171 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 00:27:45.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.385378 27171 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/data,/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/wal 00:27:45.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "11bb274b603a4ca09d30337d047b71c5" 00:27:45.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:45 on dist-test-slave-dist-test-slave-vnz7" 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.385664 27171 fs_report.cc:352] FS layout report 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/wal 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/wal 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/data/data 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 00:27:45.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 00:27:45.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.861572 27171 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 00:27:45.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.864161 27171 master_main.cc:83] Starting Master server... 00:27:45.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.924628 27171 rpc_server.cc:205] RPC server started. Bound to: 127.26.134.254:39417 00:27:45.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.925602 27171 webserver.cc:175] Starting webserver on 127.26.134.254:0 00:27:45.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.925712 27171 webserver.cc:186] Document root disabled 00:27:45.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.927300 27171 webserver.cc:313] Webserver started. Bound to: http://127.26.134.254:39885/ 00:27:45.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.928151 27171 server_base.cc:608] Dumped server information to /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-0/data/info.pb 00:27:45.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.931079 27232 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:45.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.936460 27163 external_mini_cluster.cc:995] Started /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master as pid 27171 00:27:45.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.937633 27163 external_mini_cluster.cc:933] Running /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master 00:27:45.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/wal 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/data 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.26.134.253:40601 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.26.134.253 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/logs 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/data/info.pb 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 00:27:45.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.26.134.254:39417,127.26.134.253:40601,127.26.134.252:33715 with env {} 00:27:45.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.945008 27232 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } has no permanent_uuid. Determining permanent_uuid... 00:27:45.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.977073 27232 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } has no permanent_uuid. Determining permanent_uuid... 00:27:45.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:45.978672 27232 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.253:40601: Network error: Client connection negotiation failed: client connection to 127.26.134.253:40601: connect: Connection refused (error 111) 00:27:45.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:45.998092 27232 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } attempt: 1 00:27:46.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:45.999832 27232 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.253:40601: Network error: Client connection negotiation failed: client connection to 127.26.134.253:40601: connect: Connection refused (error 111) 00:27:46.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.048180 27232 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } attempt: 2 00:27:46.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.049981 27232 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.253:40601: Network error: Client connection negotiation failed: client connection to 127.26.134.253:40601: connect: Connection refused (error 111) 00:27:46.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 00:27:46.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.091087 27233 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 00:27:46.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.091536 27233 flags.cc:406] Enabled unsafe flag: --never_fsync=true 00:27:46.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.103286 27233 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 00:27:46.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.103489 27233 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 00:27:46.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.103634 27233 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 00:27:46.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.103847 27233 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true 00:27:46.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.122181 27233 master_main.cc:73] Master server non-default flags: 00:27:46.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/data 00:27:46.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/wal 00:27:46.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 00:27:46.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.26.134.254:39417,127.26.134.253:40601,127.26.134.252:33715 00:27:46.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:46.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 00:27:46.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.26.134.253:40601 00:27:46.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 00:27:46.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 00:27:46.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:46.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/data/info.pb 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.26.134.253 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/logs 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-SNAPSHOT 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5673617937533e44afc5412722108dfed4a7f7b2 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by None at 20 Feb 2019 00:18:11 UTC on jenkins-slave-8qw8 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 16477 00:27:46.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 00:27:46.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.135947 27233 master_main.cc:80] Initializing master server... 00:27:46.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.137681 27233 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 95171us 00:27:46.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.138535 27233 fs_manager.cc:263] Metadata directory not provided 00:27:46.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.138702 27233 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 00:27:46.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.139037 27233 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 00:27:46.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.139168 27233 server_base.cc:434] Attempting to create new FS layout instead 00:27:46.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.142361 27233 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/data/instance: 00:27:46.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "6e692801b20e44569c4ca2f30f5ca7df" 00:27:46.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:46 on dist-test-slave-dist-test-slave-vnz7" 00:27:46.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.143122 27233 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/wal/instance: 00:27:46.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "6e692801b20e44569c4ca2f30f5ca7df" 00:27:46.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:46 on dist-test-slave-dist-test-slave-vnz7" 00:27:46.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.153861 27233 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.005s sys 0.004s 00:27:46.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.155218 27233 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 00:27:46.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.155740 27233 file_cache.cc:466] Constructed file cache lbm with capacity 419430 00:27:46.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.161375 27232 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } attempt: 3 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.162878 27232 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.253:40601: Network error: Client connection negotiation failed: client connection to 127.26.134.253:40601: connect: Connection refused (error 111) 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.163066 27233 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.163228 27233 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/data,/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/wal 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "6e692801b20e44569c4ca2f30f5ca7df" 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:46 on dist-test-slave-dist-test-slave-vnz7" 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.163493 27233 fs_report.cc:352] FS layout report 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/wal 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/wal 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/data/data 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 00:27:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 00:27:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.339228 27232 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } attempt: 4 00:27:46.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.341236 27232 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.253:40601: Network error: Client connection negotiation failed: client connection to 127.26.134.253:40601: connect: Connection refused (error 111) 00:27:46.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.644558 27232 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } attempt: 5 00:27:46.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.646128 27232 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.253:40601: Network error: Client connection negotiation failed: client connection to 127.26.134.253:40601: connect: Connection refused (error 111) 00:27:46.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.649571 27233 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 00:27:46.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.652592 27233 master_main.cc:83] Starting Master server... 00:27:46.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.715776 27233 rpc_server.cc:205] RPC server started. Bound to: 127.26.134.253:40601 00:27:46.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.716945 27233 webserver.cc:175] Starting webserver on 127.26.134.253:0 00:27:46.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.717064 27233 webserver.cc:186] Document root disabled 00:27:46.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.718874 27233 webserver.cc:313] Webserver started. Bound to: http://127.26.134.253:45627/ 00:27:46.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.719995 27233 server_base.cc:608] Dumped server information to /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-1/data/info.pb 00:27:46.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.723809 27296 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:46.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.729537 27163 external_mini_cluster.cc:995] Started /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master as pid 27233 00:27:46.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.730669 27163 external_mini_cluster.cc:933] Running /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master 00:27:46.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master 00:27:46.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/wal 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/data 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.26.134.252:33715 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.26.134.252 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/logs 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/data/info.pb 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 00:27:46.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.26.134.254:39417,127.26.134.253:40601,127.26.134.252:33715 with env {} 00:27:46.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.741102 27296 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } has no permanent_uuid. Determining permanent_uuid... 00:27:46.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.766876 27296 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } has no permanent_uuid. Determining permanent_uuid... 00:27:46.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.782868 27296 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } has no permanent_uuid. Determining permanent_uuid... 00:27:46.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.784451 27296 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.252:33715: Network error: Client connection negotiation failed: client connection to 127.26.134.252:33715: connect: Connection refused (error 111) 00:27:46.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.803797 27296 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } attempt: 1 00:27:46.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.805351 27296 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.252:33715: Network error: Client connection negotiation failed: client connection to 127.26.134.252:33715: connect: Connection refused (error 111) 00:27:46.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.853683 27296 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } attempt: 2 00:27:46.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.855310 27296 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.252:33715: Network error: Client connection negotiation failed: client connection to 127.26.134.252:33715: connect: Connection refused (error 111) 00:27:46.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 00:27:46.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.903604 27297 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 00:27:46.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.904053 27297 flags.cc:406] Enabled unsafe flag: --never_fsync=true 00:27:46.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.916311 27297 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 00:27:46.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.916487 27297 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 00:27:46.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.916600 27297 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 00:27:46.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.916738 27297 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.934885 27297 master_main.cc:73] Master server non-default flags: 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/data 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/wal 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.26.134.254:39417,127.26.134.253:40601,127.26.134.252:33715 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.26.134.252:33715 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/data/info.pb 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.26.134.252 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/logs 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-SNAPSHOT 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5673617937533e44afc5412722108dfed4a7f7b2 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by None at 20 Feb 2019 00:18:11 UTC on jenkins-slave-8qw8 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 16477 00:27:46.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 00:27:46.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.948360 27297 master_main.cc:80] Initializing master server... 00:27:46.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.950057 27297 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 95171us 00:27:46.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.950851 27297 fs_manager.cc:263] Metadata directory not provided 00:27:46.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.950994 27297 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 00:27:46.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.951272 27297 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 00:27:46.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.951419 27297 server_base.cc:434] Attempting to create new FS layout instead 00:27:46.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.954445 27297 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/data/instance: 00:27:46.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "de93120f73344e0b8e3fd3b24fa46e1a" 00:27:46.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:46 on dist-test-slave-dist-test-slave-vnz7" 00:27:46.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.955135 27297 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/wal/instance: 00:27:46.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "de93120f73344e0b8e3fd3b24fa46e1a" 00:27:46.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:46 on dist-test-slave-dist-test-slave-vnz7" 00:27:46.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.965466 27297 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.005s sys 0.004s 00:27:46.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.966682 27296 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } attempt: 3 00:27:46.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.966938 27297 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 00:27:46.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.967607 27297 file_cache.cc:466] Constructed file cache lbm with capacity 419430 00:27:46.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:46.968382 27296 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.252:33715: Network error: Client connection negotiation failed: client connection to 127.26.134.252:33715: connect: Connection refused (error 111) 00:27:46.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.974790 27297 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.974920 27297 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/data,/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/wal 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "de93120f73344e0b8e3fd3b24fa46e1a" 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:46 on dist-test-slave-dist-test-slave-vnz7" 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:46.975149 27297 fs_report.cc:352] FS layout report 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/wal 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/wal 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/data/data 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 00:27:46.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 00:27:47.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.144800 27296 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } attempt: 4 00:27:47.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.146523 27296 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.252:33715: Network error: Client connection negotiation failed: client connection to 127.26.134.252:33715: connect: Connection refused (error 111) 00:27:47.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.180503 27232 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } attempt: 6 00:27:47.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.192936 27232 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } has no permanent_uuid. Determining permanent_uuid... 00:27:47.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.194516 27232 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.252:33715: Network error: Client connection negotiation failed: client connection to 127.26.134.252:33715: connect: Connection refused (error 111) 00:27:47.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.226848 27232 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } attempt: 1 00:27:47.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.228574 27232 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.252:33715: Network error: Client connection negotiation failed: client connection to 127.26.134.252:33715: connect: Connection refused (error 111) 00:27:47.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.296942 27232 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } attempt: 2 00:27:47.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.298746 27232 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.252:33715: Network error: Client connection negotiation failed: client connection to 127.26.134.252:33715: connect: Connection refused (error 111) 00:27:47.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.375078 27232 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } attempt: 3 00:27:47.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.376920 27232 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.252:33715: Network error: Client connection negotiation failed: client connection to 127.26.134.252:33715: connect: Connection refused (error 111) 00:27:47.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.424263 27297 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 00:27:47.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.426714 27297 master_main.cc:83] Starting Master server... 00:27:47.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.449856 27296 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } attempt: 5 00:27:47.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.452265 27296 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.26.134.252:33715: Network error: Client connection negotiation failed: client connection to 127.26.134.252:33715: connect: Connection refused (error 111) 00:27:47.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.506074 27297 rpc_server.cc:205] RPC server started. Bound to: 127.26.134.252:33715 00:27:47.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.506990 27297 webserver.cc:175] Starting webserver on 127.26.134.252:0 00:27:47.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.507100 27297 webserver.cc:186] Document root disabled 00:27:47.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.513790 27297 webserver.cc:313] Webserver started. Bound to: http://127.26.134.252:45053/ 00:27:47.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.514261 27232 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } attempt: 4 00:27:47.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.514714 27297 server_base.cc:608] Dumped server information to /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/master-2/data/info.pb 00:27:47.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.518528 27163 external_mini_cluster.cc:995] Started /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master as pid 27297 00:27:47.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.519855 27362 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.520160 27163 external_mini_cluster.cc:933] Running /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/wal 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/data 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.26.134.193:0 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.26.134.193 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.26.134.193 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.26.134.254:39417,127.26.134.253:40601,127.26.134.252:33715 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:47.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 00:27:47.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:47.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/logs 00:27:47.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/data/info.pb 00:27:47.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:47.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 00:27:47.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 00:27:47.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags with env {} 00:27:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.541215 27362 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } has no permanent_uuid. Determining permanent_uuid... 00:27:47.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.543809 27232 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5: Bootstrap starting. 00:27:47.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.546658 27232 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5: No blocks or log segments found. Creating new log. 00:27:47.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.548121 27232 log.cc:525] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5: Log is configured to *not* fsync() on all Append() calls 00:27:47.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.552068 27232 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5: No bootstrap required, opened a new log 00:27:47.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.562829 27362 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } has no permanent_uuid. Determining permanent_uuid... 00:27:47.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.569941 27232 raft_consensus.cc:340] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } 00:27:47.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.570652 27232 raft_consensus.cc:366] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:47.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.570849 27232 raft_consensus.cc:705] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11bb274b603a4ca09d30337d047b71c5, State: Initialized, Role: FOLLOWER 00:27:47.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.571494 27232 consensus_queue.cc:229] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } 00:27:47.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.572540 27362 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } has no permanent_uuid. Determining permanent_uuid... 00:27:47.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.573937 27368 sys_catalog.cc:337] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:47.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.574658 27368 sys_catalog.cc:340] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [sys.catalog]: This master's current role is: FOLLOWER 00:27:47.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.575094 27232 sys_catalog.cc:337] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:47.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.575747 27232 sys_catalog.cc:340] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [sys.catalog]: This master's current role is: FOLLOWER 00:27:47.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.576885 27232 sys_catalog.cc:424] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [sys.catalog]: configured and running, proceeding with master startup. 00:27:47.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.578539 27369 catalog_manager.cc:529] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 00:27:47.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.579380 27171 master_main.cc:86] Master server successfully started. 00:27:47.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.586283 27362 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a: Bootstrap starting. 00:27:47.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.588788 27362 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a: No blocks or log segments found. Creating new log. 00:27:47.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.590071 27362 log.cc:525] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a: Log is configured to *not* fsync() on all Append() calls 00:27:47.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.592625 27362 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a: No bootstrap required, opened a new log 00:27:47.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.603791 27362 raft_consensus.cc:340] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } 00:27:47.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.604365 27362 raft_consensus.cc:366] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:47.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.604542 27362 raft_consensus.cc:705] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de93120f73344e0b8e3fd3b24fa46e1a, State: Initialized, Role: FOLLOWER 00:27:47.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.605015 27362 consensus_queue.cc:229] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } 00:27:47.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.606828 27362 sys_catalog.cc:337] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:47.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.608173 27362 sys_catalog.cc:340] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [sys.catalog]: This master's current role is: FOLLOWER 00:27:47.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.609581 27362 sys_catalog.cc:424] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [sys.catalog]: configured and running, proceeding with master startup. 00:27:47.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.617436 27188 tablet.cc:1786] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5: Can't schedule compaction. Clean time has not been advanced past its initial value. 00:27:47.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 00:27:47.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.693462 27364 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 00:27:47.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.694012 27364 flags.cc:406] Enabled unsafe flag: --never_fsync=true 00:27:47.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.710613 27364 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 00:27:47.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.711179 27364 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.134.193 00:27:47.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.607132 27371 sys_catalog.cc:337] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:47.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.718230 27371 sys_catalog.cc:340] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [sys.catalog]: This master's current role is: FOLLOWER 00:27:47.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.718596 27371 raft_consensus.cc:472] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 00:27:47.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.718932 27371 raft_consensus.cc:494] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } 00:27:47.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.722642 27371 leader_election.cc:296] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11bb274b603a4ca09d30337d047b71c5 (127.26.134.254:39417), 6e692801b20e44569c4ca2f30f5ca7df (127.26.134.253:40601) 00:27:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.735090 27364 tablet_server_main.cc:78] Tablet server non-default flags: 00:27:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/data 00:27:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/wal 00:27:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.26.134.193:0 00:27:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 00:27:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/data/info.pb 00:27:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.26.134.193 00:27:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:47.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.26.134.254:39417,127.26.134.253:40601,127.26.134.252:33715 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.26.134.193 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/logs 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-SNAPSHOT 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5673617937533e44afc5412722108dfed4a7f7b2 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by None at 20 Feb 2019 00:18:11 UTC on jenkins-slave-8qw8 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 16477 00:27:47.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 00:27:47.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.759312 27364 tablet_server_main.cc:85] Initializing tablet server... 00:27:47.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.762259 27364 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 95671us 00:27:47.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.762948 27364 fs_manager.cc:263] Metadata directory not provided 00:27:47.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.763218 27364 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 00:27:47.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.763682 27364 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 00:27:47.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.763923 27364 server_base.cc:434] Attempting to create new FS layout instead 00:27:47.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.768301 27364 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/data/instance: 00:27:47.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8ca697fd944c4820985a81dc9031b4d0" 00:27:47.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:47 on dist-test-slave-dist-test-slave-vnz7" 00:27:47.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.769436 27364 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/wal/instance: 00:27:47.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8ca697fd944c4820985a81dc9031b4d0" 00:27:47.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:47 on dist-test-slave-dist-test-slave-vnz7" 00:27:47.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.679152 27318 tablet.cc:1786] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a: Can't schedule compaction. Clean time has not been advanced past its initial value. 00:27:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.775699 27297 master_main.cc:86] Master server successfully started. 00:27:47.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.776553 27209 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11bb274b603a4ca09d30337d047b71c5" is_pre_election: true 00:27:47.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.777418 27209 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de93120f73344e0b8e3fd3b24fa46e1a in term 0. 00:27:47.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.778903 27309 leader_election.cc:310] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11bb274b603a4ca09d30337d047b71c5, de93120f73344e0b8e3fd3b24fa46e1a; no voters: 00:27:47.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.780562 27371 raft_consensus.cc:2640] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [term 0 FOLLOWER]: Leader pre-election won for term 1 00:27:47.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.780869 27371 raft_consensus.cc:472] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 00:27:47.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.781205 27371 raft_consensus.cc:2886] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [term 0 FOLLOWER]: Advancing to term 1 00:27:47.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.782999 27372 catalog_manager.cc:1104] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a: acquiring CA information for follower catalog manager: Not found: root CA entry not found 00:27:47.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:47.783416 27372 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 00:27:47.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.783706 27364 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.013s sys 0.000s 00:27:47.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.785749 27364 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 00:27:47.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.786423 27364 file_cache.cc:466] Constructed file cache lbm with capacity 419430 00:27:47.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.786478 27371 raft_consensus.cc:494] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } 00:27:47.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.788841 27209 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11bb274b603a4ca09d30337d047b71c5" 00:27:47.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.789336 27209 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [term 0 FOLLOWER]: Advancing to term 1 00:27:47.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.789120 27371 leader_election.cc:296] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [CANDIDATE]: Term 1 election: Requested vote from peers 11bb274b603a4ca09d30337d047b71c5 (127.26.134.254:39417), 6e692801b20e44569c4ca2f30f5ca7df (127.26.134.253:40601) 00:27:47.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.794631 27209 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de93120f73344e0b8e3fd3b24fa46e1a in term 1. 00:27:47.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.795783 27309 leader_election.cc:310] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11bb274b603a4ca09d30337d047b71c5, de93120f73344e0b8e3fd3b24fa46e1a; no voters: 00:27:47.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.796598 27371 raft_consensus.cc:2640] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [term 1 FOLLOWER]: Leader election won for term 1 00:27:47.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.797315 27371 raft_consensus.cc:667] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [term 1 LEADER]: Becoming Leader. State: Replica: de93120f73344e0b8e3fd3b24fa46e1a, State: Running, Role: LEADER 00:27:47.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.798032 27371 consensus_queue.cc:206] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } 00:27:47.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.800715 27364 fs_manager.cc:419] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s 00:27:47.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.800915 27364 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/data,/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/wal 00:27:47.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8ca697fd944c4820985a81dc9031b4d0" 00:27:47.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:47 on dist-test-slave-dist-test-slave-vnz7" 00:27:47.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.801261 27364 fs_report.cc:352] FS layout report 00:27:47.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 00:27:47.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/wal 00:27:47.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/wal 00:27:47.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/data/data 00:27:47.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 00:27:47.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 00:27:47.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 00:27:47.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 00:27:47.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 00:27:47.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 00:27:47.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 00:27:47.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 00:27:47.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 00:27:47.802 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 00:27:47.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.804747 27373 sys_catalog.cc:337] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader de93120f73344e0b8e3fd3b24fa46e1a. Latest consensus state: current_term: 1 leader_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:47.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.805759 27373 sys_catalog.cc:340] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [sys.catalog]: This master's current role is: LEADER 00:27:47.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.808576 27383 catalog_manager.cc:1046] Loading table and tablet metadata into memory... 00:27:47.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.811992 27383 catalog_manager.cc:1057] Initializing Kudu internal certificate authority... 00:27:47.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.907922 27209 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [term 1 FOLLOWER]: Refusing update from remote peer de93120f73344e0b8e3fd3b24fa46e1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 00:27:47.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.909651 27373 consensus_queue.cc:984] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 00:27:47.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.925930 27368 sys_catalog.cc:337] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de93120f73344e0b8e3fd3b24fa46e1a. Latest consensus state: current_term: 1 leader_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:47.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.926662 27368 sys_catalog.cc:340] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [sys.catalog]: This master's current role is: FOLLOWER 00:27:47.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.933595 27387 mvcc.cc:203] Tried to move safe_time back from 6351349628497334272 to 6351349628109332480. Current Snapshot: MvccSnapshot[committed={T|T < 6351349628497334272}] 00:27:47.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.936100 27373 sys_catalog.cc:337] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:47.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.936820 27373 sys_catalog.cc:340] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [sys.catalog]: This master's current role is: LEADER 00:27:47.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.940274 27368 sys_catalog.cc:337] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:47.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.940979 27368 sys_catalog.cc:340] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5 [sys.catalog]: This master's current role is: FOLLOWER 00:27:47.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.941200 27383 catalog_manager.cc:941] Generated new certificate authority record 00:27:47.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.943401 27383 catalog_manager.cc:1066] Loading token signing keys... 00:27:47.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.969241 27383 catalog_manager.cc:4095] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a: Generated new TSK 0 00:27:47.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:47.986533 27296 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } attempt: 6 00:27:48.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.006834 27296 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df: Bootstrap starting. 00:27:48.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.009640 27296 tablet_bootstrap.cc:593] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df: No blocks or log segments found. Creating new log. 00:27:48.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.011021 27296 log.cc:525] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df: Log is configured to *not* fsync() on all Append() calls 00:27:48.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.017510 27296 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df: No bootstrap required, opened a new log 00:27:48.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.036128 27296 raft_consensus.cc:340] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } 00:27:48.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.036880 27296 raft_consensus.cc:366] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:48.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.037124 27296 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e692801b20e44569c4ca2f30f5ca7df, State: Initialized, Role: FOLLOWER 00:27:48.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.037945 27296 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } 00:27:48.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.043987 27296 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:48.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.044755 27296 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [sys.catalog]: This master's current role is: FOLLOWER 00:27:48.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.045907 27296 sys_catalog.cc:424] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [sys.catalog]: configured and running, proceeding with master startup. 00:27:48.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.044507 27395 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:48.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.133989 27395 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [sys.catalog]: This master's current role is: FOLLOWER 00:27:48.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.134521 27272 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e692801b20e44569c4ca2f30f5ca7df" 00:27:48.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.135251 27272 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [term 0 FOLLOWER]: Advancing to term 1 00:27:48.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.139732 27273 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e692801b20e44569c4ca2f30f5ca7df" is_pre_election: true 00:27:48.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.142117 27233 master_main.cc:86] Master server successfully started. 00:27:48.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.142832 27272 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de93120f73344e0b8e3fd3b24fa46e1a in term 1. 00:27:48.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.143337 27273 raft_consensus.cc:2321] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de93120f73344e0b8e3fd3b24fa46e1a for term 1 because replica is already servicing an update from a current leader or another vote. 00:27:48.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.144511 27271 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [term 1 FOLLOWER]: Refusing update from remote peer de93120f73344e0b8e3fd3b24fa46e1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 00:27:48.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.147339 27371 consensus_queue.cc:984] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 00:27:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:48.155546 27252 tablet.cc:1786] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df: Can't schedule compaction. Clean time has not been advanced past its initial value. 00:27:48.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:48.179857 27396 catalog_manager.cc:1104] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df: acquiring CA information for follower catalog manager: Not found: root CA entry not found 00:27:48.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:48.180303 27396 catalog_manager.cc:587] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 00:27:48.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.205001 27395 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [sys.catalog]: SysCatalogTable state changed. Reason: New leader de93120f73344e0b8e3fd3b24fa46e1a. Latest consensus state: current_term: 1 leader_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:48.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.205678 27395 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [sys.catalog]: This master's current role is: FOLLOWER 00:27:48.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.206593 27400 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:48.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.208804 27400 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df [sys.catalog]: This master's current role is: FOLLOWER 00:27:48.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.596585 27369 catalog_manager.cc:1102] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5: acquiring CA information for follower catalog manager: success 00:27:48.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:48.601469 27364 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.766s user 0.161s sys 0.592s 00:27:48.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:48.601862 27364 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.766s user 0.161s sys 0.593s 00:27:48.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.609766 27369 catalog_manager.cc:1130] T 00000000000000000000000000000000 P 11bb274b603a4ca09d30337d047b71c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 00:27:48.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.676601 27364 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 00:27:48.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.689105 27364 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) 00:27:48.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.699681 27364 tablet_server_main.cc:90] Starting tablet server... 00:27:48.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.763145 27371 sys_catalog.cc:337] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11bb274b603a4ca09d30337d047b71c5" member_type: VOTER last_known_addr { host: "127.26.134.254" port: 39417 } } peers { permanent_uuid: "6e692801b20e44569c4ca2f30f5ca7df" member_type: VOTER last_known_addr { host: "127.26.134.253" port: 40601 } } peers { permanent_uuid: "de93120f73344e0b8e3fd3b24fa46e1a" member_type: VOTER last_known_addr { host: "127.26.134.252" port: 33715 } } } 00:27:48.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:48.766465 27371 sys_catalog.cc:340] T 00000000000000000000000000000000 P de93120f73344e0b8e3fd3b24fa46e1a [sys.catalog]: This master's current role is: LEADER 00:27:49.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.187674 27396 catalog_manager.cc:1102] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df: acquiring CA information for follower catalog manager: success 00:27:49.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.191860 27396 catalog_manager.cc:1130] T 00000000000000000000000000000000 P 6e692801b20e44569c4ca2f30f5ca7df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 00:27:49.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.291851 27364 rpc_server.cc:205] RPC server started. Bound to: 127.26.134.193:42531 00:27:49.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.293761 27364 webserver.cc:175] Starting webserver on 127.26.134.193:0 00:27:49.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.293929 27364 webserver.cc:186] Document root disabled 00:27:49.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.295883 27364 webserver.cc:313] Webserver started. Bound to: http://127.26.134.193:39911/ 00:27:49.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.296845 27364 server_base.cc:608] Dumped server information to /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-0/data/info.pb 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.301932 27163 external_mini_cluster.cc:995] Started /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver as pid 27364 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.303208 27163 external_mini_cluster.cc:933] Running /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/wal 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/data 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.26.134.194:0 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.26.134.194 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.26.134.194 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.26.134.254:39417,127.26.134.253:40601,127.26.134.252:33715 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/logs 00:27:49.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/data/info.pb 00:27:49.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:49.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 00:27:49.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 00:27:49.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags with env {} 00:27:49.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.315994 27364 tablet_server_main.cc:93] Tablet server successfully started. 00:27:49.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.365025 27515 heartbeater.cc:345] Connected to a master server at 127.26.134.252:33715 00:27:49.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.365341 27515 heartbeater.cc:426] Registering TS with master... 00:27:49.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.366008 27515 heartbeater.cc:475] Master 127.26.134.252:33715 requested a full tablet report, sending... 00:27:49.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.368732 27329 ts_manager.cc:98] Registered new tserver with Master: 8ca697fd944c4820985a81dc9031b4d0 (127.26.134.193:42531) 00:27:49.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.372283 27329 master_service.cc:269] Signed X509 certificate for tserver {username='slave'} at 127.26.134.193:55967 00:27:49.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.388777 27518 heartbeater.cc:345] Connected to a master server at 127.26.134.254:39417 00:27:49.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.389047 27518 heartbeater.cc:426] Registering TS with master... 00:27:49.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.389523 27518 heartbeater.cc:475] Master 127.26.134.254:39417 requested a full tablet report, sending... 00:27:49.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.392112 27516 heartbeater.cc:345] Connected to a master server at 127.26.134.253:40601 00:27:49.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.392374 27199 ts_manager.cc:98] Registered new tserver with Master: 8ca697fd944c4820985a81dc9031b4d0 (127.26.134.193:42531) 00:27:49.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.392424 27516 heartbeater.cc:426] Registering TS with master... 00:27:49.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.393399 27516 heartbeater.cc:475] Master 127.26.134.253:40601 requested a full tablet report, sending... 00:27:49.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.396363 27263 ts_manager.cc:98] Registered new tserver with Master: 8ca697fd944c4820985a81dc9031b4d0 (127.26.134.193:42531) 00:27:49.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 00:27:49.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:49.556041 27523 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 00:27:49.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:49.556929 27523 flags.cc:406] Enabled unsafe flag: --never_fsync=true 00:27:49.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:49.578197 27523 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 00:27:49.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:49.578963 27523 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.134.194 00:27:49.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.616436 27523 tablet_server_main.cc:78] Tablet server non-default flags: 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/data 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/wal 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.26.134.194:0 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/data/info.pb 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.26.134.194 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.26.134.254:39417,127.26.134.253:40601,127.26.134.252:33715 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.26.134.194 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/logs 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-SNAPSHOT 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5673617937533e44afc5412722108dfed4a7f7b2 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by None at 20 Feb 2019 00:18:11 UTC on jenkins-slave-8qw8 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 16477 00:27:49.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 00:27:49.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.642261 27523 tablet_server_main.cc:85] Initializing tablet server... 00:27:49.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.645541 27523 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 96671us 00:27:49.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.646288 27523 fs_manager.cc:263] Metadata directory not provided 00:27:49.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.646589 27523 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 00:27:49.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.647060 27523 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 00:27:49.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.647380 27523 server_base.cc:434] Attempting to create new FS layout instead 00:27:49.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.652360 27523 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/data/instance: 00:27:49.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "aeb753881ad146c5a9e09c756060fec8" 00:27:49.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:49 on dist-test-slave-dist-test-slave-vnz7" 00:27:49.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.653628 27523 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/wal/instance: 00:27:49.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "aeb753881ad146c5a9e09c756060fec8" 00:27:49.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:49 on dist-test-slave-dist-test-slave-vnz7" 00:27:49.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.680842 27523 fs_manager.cc:503] Time spent creating directory manager: real 0.027s user 0.010s sys 0.015s 00:27:49.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.683073 27523 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 00:27:49.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.683888 27523 file_cache.cc:466] Constructed file cache lbm with capacity 419430 00:27:49.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.696702 27523 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s 00:27:49.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.697015 27523 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/data,/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/wal 00:27:49.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "aeb753881ad146c5a9e09c756060fec8" 00:27:49.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:49 on dist-test-slave-dist-test-slave-vnz7" 00:27:49.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:49.697562 27523 fs_report.cc:352] FS layout report 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/wal 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/wal 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/data/data 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 00:27:49.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 00:27:50.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.377203 27515 heartbeater.cc:467] Master 127.26.134.252:33715 was elected leader, sending a full tablet report... 00:27:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:50.554913 27523 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.823s user 0.001s sys 0.000s 00:27:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:50.555169 27523 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.823s user 0.002s sys 0.001s 00:27:50.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.600136 27523 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 00:27:50.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.608852 27523 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) 00:27:50.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.619731 27523 tablet_server_main.cc:90] Starting tablet server... 00:27:50.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.918962 27523 rpc_server.cc:205] RPC server started. Bound to: 127.26.134.194:44099 00:27:50.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.919970 27523 webserver.cc:175] Starting webserver on 127.26.134.194:0 00:27:50.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.920083 27523 webserver.cc:186] Document root disabled 00:27:50.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.921600 27523 webserver.cc:313] Webserver started. Bound to: http://127.26.134.194:34877/ 00:27:50.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.922322 27523 server_base.cc:608] Dumped server information to /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-1/data/info.pb 00:27:50.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.927912 27163 external_mini_cluster.cc:995] Started /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver as pid 27523 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.929069 27163 external_mini_cluster.cc:933] Running /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/wal 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/data 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.26.134.195:0 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.26.134.195 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.26.134.195 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.26.134.254:39417,127.26.134.253:40601,127.26.134.252:33715 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/logs 00:27:50.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/data/info.pb 00:27:50.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:50.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 00:27:50.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 00:27:50.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags with env {} 00:27:50.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.944463 27523 tablet_server_main.cc:93] Tablet server successfully started. 00:27:50.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.963814 27646 heartbeater.cc:345] Connected to a master server at 127.26.134.252:33715 00:27:50.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.964121 27646 heartbeater.cc:426] Registering TS with master... 00:27:50.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.964790 27646 heartbeater.cc:475] Master 127.26.134.252:33715 requested a full tablet report, sending... 00:27:50.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.967525 27649 heartbeater.cc:345] Connected to a master server at 127.26.134.254:39417 00:27:50.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.967764 27649 heartbeater.cc:426] Registering TS with master... 00:27:50.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.968263 27649 heartbeater.cc:475] Master 127.26.134.254:39417 requested a full tablet report, sending... 00:27:50.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.967276 27329 ts_manager.cc:98] Registered new tserver with Master: aeb753881ad146c5a9e09c756060fec8 (127.26.134.194:44099) 00:27:50.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.970572 27329 master_service.cc:269] Signed X509 certificate for tserver {username='slave'} at 127.26.134.194:49075 00:27:50.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.970669 27199 ts_manager.cc:98] Registered new tserver with Master: aeb753881ad146c5a9e09c756060fec8 (127.26.134.194:44099) 00:27:50.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.964097 27647 heartbeater.cc:345] Connected to a master server at 127.26.134.253:40601 00:27:50.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.974864 27647 heartbeater.cc:426] Registering TS with master... 00:27:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.975245 27647 heartbeater.cc:475] Master 127.26.134.253:40601 requested a full tablet report, sending... 00:27:50.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:50.977128 27263 ts_manager.cc:98] Registered new tserver with Master: aeb753881ad146c5a9e09c756060fec8 (127.26.134.194:44099) 00:27:51.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 00:27:51.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:51.064736 27655 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 00:27:51.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:51.065225 27655 flags.cc:406] Enabled unsafe flag: --never_fsync=true 00:27:51.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:51.074523 27655 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 00:27:51.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:51.074918 27655 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.134.195 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.089416 27655 tablet_server_main.cc:78] Tablet server non-default flags: 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/data 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/wal 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.26.134.195:0 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/data/info.pb 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.26.134.195 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.26.134.254:39417,127.26.134.253:40601,127.26.134.252:33715 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 00:27:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.26.134.195 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/logs 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-SNAPSHOT 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5673617937533e44afc5412722108dfed4a7f7b2 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by None at 20 Feb 2019 00:18:11 UTC on jenkins-slave-8qw8 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 16477 00:27:51.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 00:27:51.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.103153 27655 tablet_server_main.cc:85] Initializing tablet server... 00:27:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.105167 27655 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 97671us 00:27:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.105617 27655 fs_manager.cc:263] Metadata directory not provided 00:27:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.105744 27655 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 00:27:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.105985 27655 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 00:27:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.106101 27655 server_base.cc:434] Attempting to create new FS layout instead 00:27:51.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.109066 27655 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/data/instance: 00:27:51.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "5d518f6a78594add83d76dac871d7abc" 00:27:51.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:51 on dist-test-slave-dist-test-slave-vnz7" 00:27:51.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.109756 27655 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/wal/instance: 00:27:51.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "5d518f6a78594add83d76dac871d7abc" 00:27:51.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:51 on dist-test-slave-dist-test-slave-vnz7" 00:27:51.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.119905 27655 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.001s 00:27:51.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.121179 27655 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 00:27:51.121 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.121647 27655 file_cache.cc:466] Constructed file cache lbm with capacity 419430 00:27:51.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.128707 27655 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.128834 27655 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/data,/tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/wal 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "5d518f6a78594add83d76dac871d7abc" 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-02-20 00:27:51 on dist-test-slave-dist-test-slave-vnz7" 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.129053 27655 fs_report.cc:352] FS layout report 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/wal 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/wal 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/data/data 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 00:27:51.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 00:27:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.612821 27655 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 00:27:51.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.617242 27655 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) 00:27:51.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.618577 27655 tablet_server_main.cc:90] Starting tablet server... 00:27:51.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.750798 27655 rpc_server.cc:205] RPC server started. Bound to: 127.26.134.195:41571 00:27:51.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.751865 27655 webserver.cc:175] Starting webserver on 127.26.134.195:0 00:27:51.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.752007 27655 webserver.cc:186] Document root disabled 00:27:51.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.753365 27655 webserver.cc:313] Webserver started. Bound to: http://127.26.134.195:42091/ 00:27:51.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.754104 27655 server_base.cc:608] Dumped server information to /tmp/dist-test-taskc0vaMl/test-tmp/mini-kudu-cluster2398532285586734632/ts-2/data/info.pb 00:27:51.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.763219 27163 external_mini_cluster.cc:995] Started /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver as pid 27655 00:27:51.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.777019 27655 tablet_server_main.cc:93] Tablet server successfully started. 00:27:51.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.811643 27778 heartbeater.cc:345] Connected to a master server at 127.26.134.253:40601 00:27:51.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.811987 27778 heartbeater.cc:426] Registering TS with master... 00:27:51.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.812702 27778 heartbeater.cc:475] Master 127.26.134.253:40601 requested a full tablet report, sending... 00:27:51.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.814677 27777 heartbeater.cc:345] Connected to a master server at 127.26.134.252:33715 00:27:51.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.814934 27777 heartbeater.cc:426] Registering TS with master... 00:27:51.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.815158 27263 ts_manager.cc:98] Registered new tserver with Master: 5d518f6a78594add83d76dac871d7abc (127.26.134.195:41571) 00:27:51.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.815502 27777 heartbeater.cc:475] Master 127.26.134.252:33715 requested a full tablet report, sending... 00:27:51.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.817227 27329 ts_manager.cc:98] Registered new tserver with Master: 5d518f6a78594add83d76dac871d7abc (127.26.134.195:41571) 00:27:51.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.818938 27329 master_service.cc:269] Signed X509 certificate for tserver {username='slave'} at 127.26.134.195:36857 00:27:51.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.812150 27781 heartbeater.cc:345] Connected to a master server at 127.26.134.254:39417 00:27:51.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.825106 27781 heartbeater.cc:426] Registering TS with master... 00:27:51.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.825423 27781 heartbeater.cc:475] Master 127.26.134.254:39417 requested a full tablet report, sending... 00:27:51.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.827600 27199 ts_manager.cc:98] Registered new tserver with Master: 5d518f6a78594add83d76dac871d7abc (127.26.134.195:41571) 00:27:51.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.833797 27163 external_mini_cluster.cc:623] 3 TS(s) registered with all masters 00:27:51.834 [DEBUG - main] (MiniKuduCluster.java:165) Response: 00:27:51.834 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_masters { } 00:27:51.835 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.26.134.254" port: 39417 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.26.134.253" port: 40601 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.26.134.252" port: 33715 } } } 00:27:51.836 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_tservers { } 00:27:51.837 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_tservers { tservers { id { type: TSERVER index: 0 } bound_rpc_address { host: "127.26.134.193" port: 42531 } } tservers { id { type: TSERVER index: 1 } bound_rpc_address { host: "127.26.134.194" port: 44099 } } tservers { id { type: TSERVER index: 2 } bound_rpc_address { host: "127.26.134.195" port: 41571 } } } 00:27:51.837 [INFO - main] (KuduTestHarness.java:142) Creating a new Kudu client... 00:27:51.839 [DEBUG - main] (SecurityUtil.java:123) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 00:27:51.843 [DEBUG - New I/O worker #2800] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.26.134.252 00:27:51.843 [DEBUG - New I/O worker #2802] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.26.134.254 00:27:51.843 [DEBUG - New I/O worker #2801] (Negotiator.java:460) SASL mechanism PLAIN chosen for peer 127.26.134.253 00:27:51.855 [DEBUG - New I/O worker #2800] (Negotiator.java:764) Authenticated connection [id: 0xd4ad020a, /127.0.0.1:39746 => /127.26.134.252:33715] using SASL/PLAIN 00:27:51.856 [DEBUG - New I/O worker #2801] (Negotiator.java:764) Authenticated connection [id: 0x405d3e6b, /127.0.0.1:58188 => /127.26.134.253:40601] using SASL/PLAIN 00:27:51.859 [DEBUG - New I/O worker #2800] (AsyncKuduClient.java:2118) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 00:27:51.859 [DEBUG - New I/O worker #2800] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, DeadlineTracker(timeout=50000, elapsed=19), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.26.134.252:33715, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.26.134.253:40601, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.26.134.254:39417, [19ms] Sub RPC ConnectToMaster: received response from server master-127.26.134.253:40601: OK, [19ms] Sub RPC ConnectToMaster: received response from server master-127.26.134.252:33715: OK, deferred=null) after lookup 00:27:51.870 [DEBUG - New I/O worker #2802] (Negotiator.java:764) Authenticated connection [id: 0x931135cf, /127.0.0.1:50696 => /127.26.134.254:39417] using SASL/PLAIN 00:27:51.871 [DEBUG - New I/O worker #2800] (Deferred.java:1330) callback=retry RPC@1863174929 returned Deferred@1798459944(state=PENDING, result=null, callback=(continuation of Deferred@774947114 after retry RPC@1863174929), errback=(continuation of Deferred@774947114 after retry RPC@1863174929)), so the following Deferred is getting paused: Deferred@774947114(state=PAUSED, result=Deferred@1798459944, callback=org.apache.kudu.client.AsyncKuduClient$1@1e002d4d -> org.apache.kudu.client.AsyncKuduClient$2@1fd428df -> wakeup thread Time-limited test, errback=passthrough -> passthrough -> wakeup thread Time-limited test) 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.861176 27329 catalog_manager.cc:1386] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39746: 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "testMultipleSessions" 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 00:27:51.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 00:27:51.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:51.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:51.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 00:27:51.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:51.875126 27329 catalog_manager.cc:1537] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table testMultipleSessions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 00:27:51.901 [DEBUG - New I/O worker #2800] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$1@1e002d4d@503328077 returned Deferred@1712232991(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduClient$6@4e1f3e65 -> (continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$1@1e002d4d@503328077), errback=passthrough -> (continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$1@1e002d4d@503328077)), so the following Deferred is getting paused: Deferred@774947114(state=PAUSED, result=Deferred@1712232991, callback=org.apache.kudu.client.AsyncKuduClient$2@1fd428df -> wakeup thread Time-limited test, errback=passthrough -> wakeup thread Time-limited test) 00:27:51.904 [DEBUG - New I/O worker #2800] (AsyncKuduClient.java:761) Opened table c9e6e49b83e942cba2a8c4678f5da174 00:27:51.905 [DEBUG - New I/O worker #2800] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791 returned Deferred@1374296238(state=PENDING, result=null, callback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791), errback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791)), so the following Deferred is getting paused: Deferred@774947114(state=PAUSED, result=Deferred@1374296238, callback=wakeup thread Time-limited test, errback=wakeup thread Time-limited test) 00:27:51.927 [DEBUG - New I/O worker #2800] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644 returned Deferred@1374296238(state=PENDING, result=null, callback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644), errback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644)), so the following Deferred is getting paused: Deferred@2097869081(state=PAUSED, result=Deferred@1374296238, callback=, errback=) 00:27:51.943 [DEBUG - New I/O worker #2800] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627 returned Deferred@1374296238(state=PENDING, result=null, callback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627), errback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627)), so the following Deferred is getting paused: Deferred@1275025991(state=PAUSED, result=Deferred@1374296238, callback=, errback=) 00:27:51.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.944833 27714 tablet_service.cc:870] Processing CreateTablet for tablet 50703b1ad8f94fe8a637703a5269efaf (table=testMultipleSessions [id=c9e6e49b83e942cba2a8c4678f5da174]), partition=RANGE (key) PARTITION UNBOUNDED 00:27:51.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.945595 27714 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 50703b1ad8f94fe8a637703a5269efaf. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:51.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.946130 27452 tablet_service.cc:870] Processing CreateTablet for tablet 50703b1ad8f94fe8a637703a5269efaf (table=testMultipleSessions [id=c9e6e49b83e942cba2a8c4678f5da174]), partition=RANGE (key) PARTITION UNBOUNDED 00:27:51.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.946815 27452 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 50703b1ad8f94fe8a637703a5269efaf. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:51.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.958729 27583 tablet_service.cc:870] Processing CreateTablet for tablet 50703b1ad8f94fe8a637703a5269efaf (table=testMultipleSessions [id=c9e6e49b83e942cba2a8c4678f5da174]), partition=RANGE (key) PARTITION UNBOUNDED 00:27:51.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.959697 27583 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 50703b1ad8f94fe8a637703a5269efaf. 1 dirs total, 0 dirs full, 0 dirs failed 00:27:51.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.959877 27714 ts_tablet_manager.cc:1175] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc: Registered tablet (data state: TABLET_DATA_READY) 00:27:51.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.960168 27452 ts_tablet_manager.cc:1175] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0: Registered tablet (data state: TABLET_DATA_READY) 00:27:51.962 [DEBUG - New I/O worker #2800] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756 returned Deferred@1374296238(state=PENDING, result=null, callback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627) -> (continuation of Deferred@1019775208 after org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756), errback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627) -> (continuation of Deferred@1019775208 after org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756)), so the following Deferred is getting paused: Deferred@1019775208(state=PAUSED, result=Deferred@1374296238, callback=, errback=) 00:27:51.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.961917 27819 ts_tablet_manager.cc:1030] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc: Bootstrapping tablet 00:27:51.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.962419 27819 tablet_bootstrap.cc:439] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc: Bootstrap starting. 00:27:51.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.966751 27819 tablet_bootstrap.cc:593] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc: No blocks or log segments found. Creating new log. 00:27:51.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.968103 27818 ts_tablet_manager.cc:1030] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0: Bootstrapping tablet 00:27:51.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.968744 27818 tablet_bootstrap.cc:439] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0: Bootstrap starting. 00:27:51.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.969009 27819 log.cc:525] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc: Log is configured to *not* fsync() on all Append() calls 00:27:51.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.972460 27818 tablet_bootstrap.cc:593] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0: No blocks or log segments found. Creating new log. 00:27:51.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.972931 27819 tablet_bootstrap.cc:439] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc: No bootstrap required, opened a new log 00:27:51.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.973680 27819 ts_tablet_manager.cc:1047] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s 00:27:51.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.974900 27646 heartbeater.cc:467] Master 127.26.134.252:33715 was elected leader, sending a full tablet report... 00:27:51.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.975338 27818 log.cc:525] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0: Log is configured to *not* fsync() on all Append() calls 00:27:51.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.979585 27583 ts_tablet_manager.cc:1175] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8: Registered tablet (data state: TABLET_DATA_READY) 00:27:51.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.981941 27818 tablet_bootstrap.cc:439] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0: No bootstrap required, opened a new log 00:27:51.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.982512 27818 ts_tablet_manager.cc:1047] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s 00:27:51.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.982837 27822 ts_tablet_manager.cc:1030] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8: Bootstrapping tablet 00:27:51.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.983747 27822 tablet_bootstrap.cc:439] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8: Bootstrap starting. 00:27:51.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.991628 27822 tablet_bootstrap.cc:593] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8: No blocks or log segments found. Creating new log. 00:27:51.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.993480 27822 log.cc:525] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8: Log is configured to *not* fsync() on all Append() calls 00:27:51.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.997427 27822 tablet_bootstrap.cc:439] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8: No bootstrap required, opened a new log 00:27:51.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:51.997879 27822 ts_tablet_manager.cc:1047] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s 00:27:52.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.000954 27819 raft_consensus.cc:340] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d518f6a78594add83d76dac871d7abc" member_type: VOTER last_known_addr { host: "127.26.134.195" port: 41571 } } peers { permanent_uuid: "aeb753881ad146c5a9e09c756060fec8" member_type: VOTER last_known_addr { host: "127.26.134.194" port: 44099 } } peers { permanent_uuid: "8ca697fd944c4820985a81dc9031b4d0" member_type: VOTER last_known_addr { host: "127.26.134.193" port: 42531 } } 00:27:52.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.001889 27819 raft_consensus.cc:366] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:52.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.002110 27819 raft_consensus.cc:705] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d518f6a78594add83d76dac871d7abc, State: Initialized, Role: FOLLOWER 00:27:52.002 [DEBUG - New I/O worker #2800] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@7efda4e6@2130552038 returned Deferred@1374296238(state=PENDING, result=null, callback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627) -> (continuation of Deferred@1019775208 after org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756) -> (continuation of Deferred@1331815452 after org.apache.kudu.client.AsyncKuduClient$15@7efda4e6@2130552038), errback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627) -> (continuation of Deferred@1019775208 after org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756) -> (continuation of Deferred@1331815452 after org.apache.kudu.client.AsyncKuduClient$15@7efda4e6@2130552038)), so the following Deferred is getting paused: Deferred@1331815452(state=PAUSED, result=Deferred@1374296238, callback=, errback=) 00:27:52.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.006809 27818 raft_consensus.cc:340] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d518f6a78594add83d76dac871d7abc" member_type: VOTER last_known_addr { host: "127.26.134.195" port: 41571 } } peers { permanent_uuid: "aeb753881ad146c5a9e09c756060fec8" member_type: VOTER last_known_addr { host: "127.26.134.194" port: 44099 } } peers { permanent_uuid: "8ca697fd944c4820985a81dc9031b4d0" member_type: VOTER last_known_addr { host: "127.26.134.193" port: 42531 } } 00:27:52.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.007593 27819 consensus_queue.cc:229] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d518f6a78594add83d76dac871d7abc" member_type: VOTER last_known_addr { host: "127.26.134.195" port: 41571 } } peers { permanent_uuid: "aeb753881ad146c5a9e09c756060fec8" member_type: VOTER last_known_addr { host: "127.26.134.194" port: 44099 } } peers { permanent_uuid: "8ca697fd944c4820985a81dc9031b4d0" member_type: VOTER last_known_addr { host: "127.26.134.193" port: 42531 } } 00:27:52.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.008074 27818 raft_consensus.cc:366] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:52.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.008267 27818 raft_consensus.cc:705] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca697fd944c4820985a81dc9031b4d0, State: Initialized, Role: FOLLOWER 00:27:52.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.008757 27818 consensus_queue.cc:229] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d518f6a78594add83d76dac871d7abc" member_type: VOTER last_known_addr { host: "127.26.134.195" port: 41571 } } peers { permanent_uuid: "aeb753881ad146c5a9e09c756060fec8" member_type: VOTER last_known_addr { host: "127.26.134.194" port: 44099 } } peers { permanent_uuid: "8ca697fd944c4820985a81dc9031b4d0" member_type: VOTER last_known_addr { host: "127.26.134.193" port: 42531 } } 00:27:52.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.009971 27777 heartbeater.cc:467] Master 127.26.134.252:33715 was elected leader, sending a full tablet report... 00:27:52.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.015200 27818 ts_tablet_manager.cc:1075] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0: Time spent starting tablet: real 0.032s user 0.018s sys 0.010s 00:27:52.022 [DEBUG - New I/O worker #2800] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@589e04e5@1486750949 returned Deferred@1374296238(state=PENDING, result=null, callback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627) -> (continuation of Deferred@1019775208 after org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756) -> (continuation of Deferred@1331815452 after org.apache.kudu.client.AsyncKuduClient$15@7efda4e6@2130552038) -> (continuation of Deferred@543774934 after org.apache.kudu.client.AsyncKuduClient$15@589e04e5@1486750949), errback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627) -> (continuation of Deferred@1019775208 after org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756) -> (continuation of Deferred@1331815452 after org.apache.kudu.client.AsyncKuduClient$15@7efda4e6@2130552038) -> (continuation of Deferred@543774934 after org.apache.kudu.client.AsyncKuduClient$15@589e04e5@1486750949)), so the following Deferred is getting paused: Deferred@543774934(state=PAUSED, result=Deferred@1374296238, callback=, errback=) 00:27:52.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.024235 27819 ts_tablet_manager.cc:1075] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc: Time spent starting tablet: real 0.050s user 0.024s sys 0.000s 00:27:52.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:52.028012 27783 tablet.cc:1786] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc: Can't schedule compaction. Clean time has not been advanced past its initial value. 00:27:52.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.032451 27822 raft_consensus.cc:340] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d518f6a78594add83d76dac871d7abc" member_type: VOTER last_known_addr { host: "127.26.134.195" port: 41571 } } peers { permanent_uuid: "aeb753881ad146c5a9e09c756060fec8" member_type: VOTER last_known_addr { host: "127.26.134.194" port: 44099 } } peers { permanent_uuid: "8ca697fd944c4820985a81dc9031b4d0" member_type: VOTER last_known_addr { host: "127.26.134.193" port: 42531 } } 00:27:52.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.033069 27822 raft_consensus.cc:366] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 00:27:52.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.033231 27822 raft_consensus.cc:705] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeb753881ad146c5a9e09c756060fec8, State: Initialized, Role: FOLLOWER 00:27:52.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.033675 27822 consensus_queue.cc:229] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d518f6a78594add83d76dac871d7abc" member_type: VOTER last_known_addr { host: "127.26.134.195" port: 41571 } } peers { permanent_uuid: "aeb753881ad146c5a9e09c756060fec8" member_type: VOTER last_known_addr { host: "127.26.134.194" port: 44099 } } peers { permanent_uuid: "8ca697fd944c4820985a81dc9031b4d0" member_type: VOTER last_known_addr { host: "127.26.134.193" port: 42531 } } 00:27:52.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.036620 27822 ts_tablet_manager.cc:1075] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8: Time spent starting tablet: real 0.039s user 0.017s sys 0.003s 00:27:52.062 [DEBUG - New I/O worker #2800] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@332002a9@857735849 returned Deferred@1374296238(state=PENDING, result=null, callback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627) -> (continuation of Deferred@1019775208 after org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756) -> (continuation of Deferred@1331815452 after org.apache.kudu.client.AsyncKuduClient$15@7efda4e6@2130552038) -> (continuation of Deferred@543774934 after org.apache.kudu.client.AsyncKuduClient$15@589e04e5@1486750949) -> (continuation of Deferred@1939947989 after org.apache.kudu.client.AsyncKuduClient$15@332002a9@857735849), errback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627) -> (continuation of Deferred@1019775208 after org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756) -> (continuation of Deferred@1331815452 after org.apache.kudu.client.AsyncKuduClient$15@7efda4e6@2130552038) -> (continuation of Deferred@543774934 after org.apache.kudu.client.AsyncKuduClient$15@589e04e5@1486750949) -> (continuation of Deferred@1939947989 after org.apache.kudu.client.AsyncKuduClient$15@332002a9@857735849)), so the following Deferred is getting paused: Deferred@1939947989(state=PAUSED, result=Deferred@1374296238, callback=, errback=) 00:27:52.082 [DEBUG - New I/O worker #2800] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@5b30ca68@1529924200 returned Deferred@1374296238(state=PENDING, result=null, callback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627) -> (continuation of Deferred@1019775208 after org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756) -> (continuation of Deferred@1331815452 after org.apache.kudu.client.AsyncKuduClient$15@7efda4e6@2130552038) -> (continuation of Deferred@543774934 after org.apache.kudu.client.AsyncKuduClient$15@589e04e5@1486750949) -> (continuation of Deferred@1939947989 after org.apache.kudu.client.AsyncKuduClient$15@332002a9@857735849) -> (continuation of Deferred@829789744 after org.apache.kudu.client.AsyncKuduClient$15@5b30ca68@1529924200), errback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627) -> (continuation of Deferred@1019775208 after org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756) -> (continuation of Deferred@1331815452 after org.apache.kudu.client.AsyncKuduClient$15@7efda4e6@2130552038) -> (continuation of Deferred@543774934 after org.apache.kudu.client.AsyncKuduClient$15@589e04e5@1486750949) -> (continuation of Deferred@1939947989 after org.apache.kudu.client.AsyncKuduClient$15@332002a9@857735849) -> (continuation of Deferred@829789744 after org.apache.kudu.client.AsyncKuduClient$15@5b30ca68@1529924200)), so the following Deferred is getting paused: Deferred@829789744(state=PAUSED, result=Deferred@1374296238, callback=, errback=) 00:27:52.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.113121 27825 raft_consensus.cc:472] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 00:27:52.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.113513 27825 raft_consensus.cc:494] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d518f6a78594add83d76dac871d7abc" member_type: VOTER last_known_addr { host: "127.26.134.195" port: 41571 } } peers { permanent_uuid: "aeb753881ad146c5a9e09c756060fec8" member_type: VOTER last_known_addr { host: "127.26.134.194" port: 44099 } } peers { permanent_uuid: "8ca697fd944c4820985a81dc9031b4d0" member_type: VOTER last_known_addr { host: "127.26.134.193" port: 42531 } } 00:27:52.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.115262 27825 leader_election.cc:296] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d518f6a78594add83d76dac871d7abc (127.26.134.195:41571), aeb753881ad146c5a9e09c756060fec8 (127.26.134.194:44099) 00:27:52.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:52.070361 27520 tablet.cc:1786] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0: Can't schedule compaction. Clean time has not been advanced past its initial value. 00:27:52.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.162340 27603 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "50703b1ad8f94fe8a637703a5269efaf" candidate_uuid: "8ca697fd944c4820985a81dc9031b4d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb753881ad146c5a9e09c756060fec8" is_pre_election: true 00:27:52.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.162847 27603 raft_consensus.cc:2346] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca697fd944c4820985a81dc9031b4d0 in term 0. 00:27:52.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.164212 27386 leader_election.cc:310] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ca697fd944c4820985a81dc9031b4d0, aeb753881ad146c5a9e09c756060fec8; no voters: 00:27:52.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.164906 27825 raft_consensus.cc:2640] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [term 0 FOLLOWER]: Leader pre-election won for term 1 00:27:52.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.164780 27734 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "50703b1ad8f94fe8a637703a5269efaf" candidate_uuid: "8ca697fd944c4820985a81dc9031b4d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d518f6a78594add83d76dac871d7abc" is_pre_election: true 00:27:52.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.165117 27825 raft_consensus.cc:472] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 00:27:52.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.165298 27825 raft_consensus.cc:2886] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [term 0 FOLLOWER]: Advancing to term 1 00:27:52.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.165385 27734 raft_consensus.cc:2346] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca697fd944c4820985a81dc9031b4d0 in term 0. 00:27:52.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.168943 27825 raft_consensus.cc:494] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d518f6a78594add83d76dac871d7abc" member_type: VOTER last_known_addr { host: "127.26.134.195" port: 41571 } } peers { permanent_uuid: "aeb753881ad146c5a9e09c756060fec8" member_type: VOTER last_known_addr { host: "127.26.134.194" port: 44099 } } peers { permanent_uuid: "8ca697fd944c4820985a81dc9031b4d0" member_type: VOTER last_known_addr { host: "127.26.134.193" port: 42531 } } 00:27:52.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.170120 27825 leader_election.cc:296] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [CANDIDATE]: Term 1 election: Requested vote from peers 5d518f6a78594add83d76dac871d7abc (127.26.134.195:41571), aeb753881ad146c5a9e09c756060fec8 (127.26.134.194:44099) 00:27:52.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.170941 27734 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "50703b1ad8f94fe8a637703a5269efaf" candidate_uuid: "8ca697fd944c4820985a81dc9031b4d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d518f6a78594add83d76dac871d7abc" 00:27:52.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.171031 27603 tablet_service.cc:1122] Received RequestConsensusVote() RPC: tablet_id: "50703b1ad8f94fe8a637703a5269efaf" candidate_uuid: "8ca697fd944c4820985a81dc9031b4d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aeb753881ad146c5a9e09c756060fec8" 00:27:52.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.171274 27734 raft_consensus.cc:2886] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc [term 0 FOLLOWER]: Advancing to term 1 00:27:52.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.171550 27603 raft_consensus.cc:2886] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8 [term 0 FOLLOWER]: Advancing to term 1 00:27:52.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.174450 27734 raft_consensus.cc:2346] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ca697fd944c4820985a81dc9031b4d0 in term 1. 00:27:52.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.174928 27603 raft_consensus.cc:2346] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ca697fd944c4820985a81dc9031b4d0 in term 1. 00:27:52.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.175540 27384 leader_election.cc:310] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d518f6a78594add83d76dac871d7abc, 8ca697fd944c4820985a81dc9031b4d0; no voters: 00:27:52.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.176273 27825 raft_consensus.cc:2640] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [term 1 FOLLOWER]: Leader election won for term 1 00:27:52.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.177736 27825 raft_consensus.cc:667] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [term 1 LEADER]: Becoming Leader. State: Replica: 8ca697fd944c4820985a81dc9031b4d0, State: Running, Role: LEADER 00:27:52.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.178314 27825 consensus_queue.cc:206] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d518f6a78594add83d76dac871d7abc" member_type: VOTER last_known_addr { host: "127.26.134.195" port: 41571 } } peers { permanent_uuid: "aeb753881ad146c5a9e09c756060fec8" member_type: VOTER last_known_addr { host: "127.26.134.194" port: 44099 } } peers { permanent_uuid: "8ca697fd944c4820985a81dc9031b4d0" member_type: VOTER last_known_addr { host: "127.26.134.193" port: 42531 } } 00:27:52.183 [DEBUG - New I/O worker #2800] (Deferred.java:1330) callback=org.apache.kudu.client.AsyncKuduClient$15@273effd@41152509 returned Deferred@1374296238(state=PENDING, result=null, callback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627) -> (continuation of Deferred@1019775208 after org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756) -> (continuation of Deferred@1331815452 after org.apache.kudu.client.AsyncKuduClient$15@7efda4e6@2130552038) -> (continuation of Deferred@543774934 after org.apache.kudu.client.AsyncKuduClient$15@589e04e5@1486750949) -> (continuation of Deferred@1939947989 after org.apache.kudu.client.AsyncKuduClient$15@332002a9@857735849) -> (continuation of Deferred@829789744 after org.apache.kudu.client.AsyncKuduClient$15@5b30ca68@1529924200) -> (continuation of Deferred@2066065807 after org.apache.kudu.client.AsyncKuduClient$15@273effd@41152509), errback=(continuation of Deferred@774947114 after org.apache.kudu.client.AsyncKuduClient$2@1fd428df@533997791) -> (continuation of Deferred@2097869081 after org.apache.kudu.client.AsyncKuduClient$15@6fc6409c@1875263644) -> (continuation of Deferred@1275025991 after org.apache.kudu.client.AsyncKuduClient$15@4a5104cb@1246823627) -> (continuation of Deferred@1019775208 after org.apache.kudu.client.AsyncKuduClient$15@16603104@375402756) -> (continuation of Deferred@1331815452 after org.apache.kudu.client.AsyncKuduClient$15@7efda4e6@2130552038) -> (continuation of Deferred@543774934 after org.apache.kudu.client.AsyncKuduClient$15@589e04e5@1486750949) -> (continuation of Deferred@1939947989 after org.apache.kudu.client.AsyncKuduClient$15@332002a9@857735849) -> (continuation of Deferred@829789744 after org.apache.kudu.client.AsyncKuduClient$15@5b30ca68@1529924200) -> (continuation of Deferred@2066065807 after org.apache.kudu.client.AsyncKuduClient$15@273effd@41152509)), so the following Deferred is getting paused: Deferred@2066065807(state=PAUSED, result=Deferred@1374296238, callback=, errback=) 00:27:52.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.185572 27327 catalog_manager.cc:3791] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 reported cstate change: term changed from 0 to 1, leader changed from to 8ca697fd944c4820985a81dc9031b4d0 (127.26.134.193). New cstate: current_term: 1 leader_uuid: "8ca697fd944c4820985a81dc9031b4d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d518f6a78594add83d76dac871d7abc" member_type: VOTER last_known_addr { host: "127.26.134.195" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aeb753881ad146c5a9e09c756060fec8" member_type: VOTER last_known_addr { host: "127.26.134.194" port: 44099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ca697fd944c4820985a81dc9031b4d0" member_type: VOTER last_known_addr { host: "127.26.134.193" port: 42531 } health_report { overall_health: HEALTHY } } } 00:27:52.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:27:52.195412 27656 tablet.cc:1786] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8: Can't schedule compaction. Clean time has not been advanced past its initial value. 00:27:52.269 [DEBUG - New I/O worker #2800] (AsyncKuduClient.java:2118) Learned about tablet 50703b1ad8f94fe8a637703a5269efaf for table 'testMultipleSessions' with partition [, ) 00:27:52.270 [DEBUG - New I/O worker #2800] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299999, tablet=50703b1ad8f94fe8a637703a5269efaf@[5d518f6a78594add83d76dac871d7abc(127.26.134.195:41571),8ca697fd944c4820985a81dc9031b4d0(127.26.134.193:42531)[L],aeb753881ad146c5a9e09c756060fec8(127.26.134.194:44099)]}] 00:27:52.270 [DEBUG - New I/O worker #2800] (AsyncKuduClient.java:1277) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, DeadlineTracker(timeout=30000, elapsed=3), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC GetTableLocations: sending RPC to server master-127.26.134.252:33715, [2ms] Sub RPC GetTableLocations: received response from server master-127.26.134.252:33715: OK, deferred=null) after lookup 00:27:52.271 [DEBUG - New I/O worker #2800] (Deferred.java:1330) callback=retry RPC@994220524 returned Deferred@651400574(state=PENDING, result=null, callback=(continuation of Deferred@1886140922 after retry RPC@994220524), errback=(continuation of Deferred@1886140922 after retry RPC@994220524)), so the following Deferred is getting paused: Deferred@1886140922(state=PAUSED, result=Deferred@651400574, callback=org.apache.kudu.client.AsyncKuduSession$2@cf3c3e9 -> passthrough -> wakeup thread pool-626-thread-1, errback=passthrough -> org.apache.kudu.client.AsyncKuduSession$SingleOperationErrCallback@2351b662 -> wakeup thread pool-626-thread-1) 00:27:52.288 [DEBUG - New I/O worker #2803] (Negotiator.java:764) Authenticated connection [id: 0xbf1aec20, /127.0.0.1:59704 => /127.26.134.193:42531] using TOKEN/null 00:27:52.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.304296 27603 raft_consensus.cc:1181] T 50703b1ad8f94fe8a637703a5269efaf P aeb753881ad146c5a9e09c756060fec8 [term 1 FOLLOWER]: Refusing update from remote peer 8ca697fd944c4820985a81dc9031b4d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 00:27:52.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.304796 27734 raft_consensus.cc:1181] T 50703b1ad8f94fe8a637703a5269efaf P 5d518f6a78594add83d76dac871d7abc [term 1 FOLLOWER]: Refusing update from remote peer 8ca697fd944c4820985a81dc9031b4d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 00:27:52.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.306483 27829 consensus_queue.cc:984] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeb753881ad146c5a9e09c756060fec8" member_type: VOTER last_known_addr { host: "127.26.134.194" port: 44099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s 00:27:52.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.323247 27825 consensus_queue.cc:984] T 50703b1ad8f94fe8a637703a5269efaf P 8ca697fd944c4820985a81dc9031b4d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d518f6a78594add83d76dac871d7abc" member_type: VOTER last_known_addr { host: "127.26.134.195" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s 00:27:52.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.381023 27897 mvcc.cc:203] Tried to move safe_time back from 6351349646544809984 to 6351349646047391744. Current Snapshot: MvccSnapshot[committed={T|T < 6351349646544809984}] 00:27:52.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:27:52.585117 27891 mvcc.cc:203] Tried to move safe_time back from 6351349647666425856 to 6351349646047391744. Current Snapshot: MvccSnapshot[committed={T|T < 6351349647666425856 or (T in {6351349647666425856})}] 00:28:01.861 [DEBUG - New I/O worker #2801] (Connection.java:429) [peer master-127.26.134.253:40601(127.26.134.253:40601)] encountered a read timeout; closing the channel 00:28:01.861 [DEBUG - New I/O worker #2801] (Connection.java:688) [peer master-127.26.134.253:40601(127.26.134.253:40601)] cleaning up while in state READY due to: [peer master-127.26.134.253:40601(127.26.134.253:40601)] encountered a read timeout; closing the channel 00:28:01.882 [DEBUG - New I/O worker #2802] (Connection.java:429) [peer master-127.26.134.254:39417(127.26.134.254:39417)] encountered a read timeout; closing the channel 00:28:01.882 [DEBUG - New I/O worker #2802] (Connection.java:688) [peer master-127.26.134.254:39417(127.26.134.254:39417)] cleaning up while in state READY due to: [peer master-127.26.134.254:39417(127.26.134.254:39417)] encountered a read timeout; closing the channel 00:28:02.281 [DEBUG - New I/O worker #2800] (Connection.java:429) [peer master-127.26.134.252:33715(127.26.134.252:33715)] encountered a read timeout; closing the channel 00:28:02.282 [DEBUG - New I/O worker #2800] (Connection.java:688) [peer master-127.26.134.252:33715(127.26.134.252:33715)] cleaning up while in state READY due to: [peer master-127.26.134.252:33715(127.26.134.252:33715)] encountered a read timeout; closing the channel 00:28:23.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:23.025640 27184 debug-util.cc:397] Leaking SignalData structure 0x7b080002c060 after lost signal to thread 27172 00:28:23.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:23.026259 27184 debug-util.cc:397] Leaking SignalData structure 0x7b080006e800 after lost signal to thread 27230 00:28:27.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:27.598317 27538 debug-util.cc:397] Leaking SignalData structure 0x7b080002a0a0 after lost signal to thread 27526 00:28:27.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:27.599301 27538 debug-util.cc:397] Leaking SignalData structure 0x7b080009b960 after lost signal to thread 27644 00:28:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:41.411967 27248 debug-util.cc:397] Leaking SignalData structure 0x7b080002c060 after lost signal to thread 27236 00:28:41.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:41.412575 27248 debug-util.cc:397] Leaking SignalData structure 0x7b080006e400 after lost signal to thread 27294 00:28:45.993 [INFO - Time-limited test] (ITClientStress.java:177) Upserted 6100 rows 00:28:45.994 [DEBUG - main] (Deferred.java:1330) callback=disconnect callback@1049817027 returned Deferred@23211803(state=PENDING, result=null, callback=release resources callback -> (continuation of Deferred@1923598304 after disconnect callback@1049817027), errback=passthrough -> (continuation of Deferred@1923598304 after disconnect callback@1049817027)), so the following Deferred is getting paused: Deferred@1923598304(state=PAUSED, result=Deferred@23211803, callback=, errback=) 00:28:45.994 [DEBUG - New I/O worker #2803] (AsyncKuduClient.java:2285) Releasing all remaining resources 00:28:45.995 [DEBUG - New I/O worker #2803] (Connection.java:688) [peer 8ca697fd944c4820985a81dc9031b4d0(127.26.134.193:42531)] cleaning up while in state READY due to: connection disconnected 00:28:45.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:28:45.996259 27163 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver with pid 27364 00:28:46.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.034632 27244 connection.cc:647] server connection from 127.26.134.193:59769 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.193:59769): Connection reset by peer (error 104) 00:28:46.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.034636 27183 connection.cc:647] server connection from 127.26.134.193:58207 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.193:58207): Connection reset by peer (error 104) 00:28:46.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.034647 27668 connection.cc:647] server connection from 127.26.134.193:46923 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.193:46923): Connection reset by peer (error 104) 00:28:46.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.034677 27310 connection.cc:647] client connection to 127.26.134.193:42531 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.193:42531): Connection reset by peer (error 104) 00:28:46.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.035634 27313 connection.cc:647] server connection from 127.26.134.193:55967 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.193:55967): Connection reset by peer (error 104) 00:28:46.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:28:46.036366 27163 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver with pid 27523 00:28:46.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.072924 27313 connection.cc:647] server connection from 127.26.134.194:49075 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.194:49075): Connection reset by peer (error 104) 00:28:46.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.072993 27181 connection.cc:647] server connection from 127.26.134.194:40229 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.194:40229): Connection reset by peer (error 104) 00:28:46.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.072978 27243 connection.cc:647] server connection from 127.26.134.194:33369 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.194:33369): Connection reset by peer (error 104) 00:28:46.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.073766 27310 connection.cc:647] client connection to 127.26.134.194:44099 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.194:44099): Connection reset by peer (error 104) 00:28:46.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:28:46.074009 27163 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-tserver with pid 27655 00:28:46.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.108721 27183 connection.cc:647] server connection from 127.26.134.195:53519 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.195:53519): Connection reset by peer (error 104) 00:28:46.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.108736 27243 connection.cc:647] server connection from 127.26.134.195:55239 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.195:55239): Connection reset by peer (error 104) 00:28:46.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.108817 27309 connection.cc:647] server connection from 127.26.134.195:36857 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.195:36857): Connection reset by peer (error 104) 00:28:46.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.109619 27308 connection.cc:647] client connection to 127.26.134.195:41571 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.195:41571): Connection reset by peer (error 104) 00:28:46.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:28:46.109887 27163 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master with pid 27171 00:28:46.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.134223 27308 connection.cc:647] server connection from 127.0.0.1:39730 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39730): Connection reset by peer (error 104) 00:28:46.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.134313 27245 connection.cc:647] server connection from 127.0.0.1:58158 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:58158): Connection reset by peer (error 104) 00:28:46.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.135010 27244 connection.cc:647] client connection to 127.26.134.254:39417 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.254:39417): Connection reset by peer (error 104) 00:28:46.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.135576 27309 connection.cc:647] client connection to 127.26.134.254:39417 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.254:39417): Connection reset by peer (error 104) 00:28:46.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:28:46.136392 27163 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master with pid 27233 00:28:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.162765 27313 connection.cc:647] client connection to 127.26.134.253:40601 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.253:40601): Connection reset by peer (error 104) 00:28:46.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0220 00:28:46.163413 27163 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskc0vaMl/build/tsan/bin/kudu-master with pid 27297 00:28:46.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0220 00:28:46.134426 27170 connection.cc:647] client connection to 127.26.134.254:39417 recv error: Network error: failed to read from TLS socket (remote: 127.26.134.254:39417): Connection reset by peer (error 104) Time: 84.71 OK (2 tests)