JUnit version 4.12 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/tmp/dist-test-taskusvpEt/java/build/jars/log4j-slf4j-impl-2.11.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/dist-test-taskusvpEt/java/build/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/dist-test-taskusvpEt/java/build/jars/logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 04:08:46.962 [INFO - main] (RetryRule.java:136) Creating RetryStatement with result reporter and retry count of 0 (no retries) .04:08:47.004 [INFO - Time-limited test] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskusvpEt/test-tmp 04:08:47.015 [INFO - Time-limited test] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin 04:08:47.018 [INFO - Time-limited test] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskusvpEt/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb] 04:08:47.041 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 0 enable_kerberos: true cluster_root: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457" extra_master_flags: "--authn_token_validity_seconds=10" hms_mode: NONE mini_kdc_options { ticket_lifetime: "10s" renew_lifetime: "20s" } } 04:08:47.221 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:08:47.229 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_cluster { } 04:08:47.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Loading random data 04:08:47.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Initializing database '/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/principal' for realm 'KRBTEST.COM', 04:08:47.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) master key name 'K/M@KRBTEST.COM' 04:08:47.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): setting up network... 04:08:47.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) 04:08:47.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked 04:08:47.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 04:08:47.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): skipping unrecognized local address family 17 04:08:47.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): skipping unrecognized local address family 17 04:08:47.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): set up 1 sockets 04:08:47.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): commencing operation 04:08:47.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: starting... 04:08:47.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:47.811322 809 mini_kdc.cc:119] Time spent starting KDC: real 0.555s user 0.004s sys 0.015s 04:08:47.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy 04:08:47.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:08:47.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-admin@KRBTEST.COM" created. 04:08:47.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy 04:08:47.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:08:47.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-user@KRBTEST.COM" created. 04:08:47.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy 04:08:47.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:08:47.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "joe-interloper@KRBTEST.COM" created. 04:08:47.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:08:47.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/test-user.keytab. 04:08:47.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/test-user.keytab. 04:08:47.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/test-user.keytab. 04:08:47.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/test-user.keytab. 04:08:47.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:08:47.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:08:47.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.0.202.126@KRBTEST.COM; defaulting to no policy 04:08:47.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:08:47.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.0.202.126@KRBTEST.COM" created. 04:08:47.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:08:47.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.0.202.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.126.keytab. 04:08:47.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.0.202.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.126.keytab. 04:08:47.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.0.202.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.126.keytab. 04:08:47.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.0.202.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.126.keytab. 04:08:47.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:47.962992 809 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:08:47.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:08:47.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal 04:08:47.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data 04:08:47.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:08:47.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:08:47.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:08:47.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:08:47.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.0.202.126:46581 04:08:47.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.0.202.126 04:08:47.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:08:47.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:08:47.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:08:47.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:08:47.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:08:47.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:08:47.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:08:47.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:08:47.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/logs 04:08:47.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data/info.pb 04:08:47.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:08:47.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:08:47.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:08:47.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:08:47.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.126.keytab 04:08:47.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.0.202.126 04:08:47.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:08:47.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:08:47.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:08:47.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:08:47.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:08:47.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.0.202.126:46581,127.0.202.125:43873,127.0.202.124:43241 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:08:48.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:08:48.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:48.121915 829 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:08:48.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:48.122349 829 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.0.202.126 04:08:48.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:48.122592 829 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:08:48.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:48.134333 829 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:08:48.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:48.134527 829 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:08:48.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:48.134651 829 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:08:48.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:48.134748 829 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:08:48.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:48.134865 829 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:08:48.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:48.134989 829 flags.cc:404] Enabled experimental flag: --principal=kudu/127.0.202.126 04:08:48.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.154659 829 master_main.cc:98] Master server non-default flags: 04:08:48.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data 04:08:48.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal 04:08:48.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:08:48.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:08:48.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.0.202.126:46581,127.0.202.125:43873,127.0.202.124:43241 04:08:48.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:08:48.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.0.202.126:46581 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.126.keytab 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.0.202.126 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data/info.pb 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.0.202.126 04:08:48.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/logs 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:08:48.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:08:48.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:08:48.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.171002 829 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 101016us 04:08:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:48 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:08:48.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.173606 829 init.cc:359] Logged in from keytab as kudu/127.0.202.126@KRBTEST.COM (short username kudu) 04:08:48.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.176856 829 fs_manager.cc:263] Metadata directory not provided 04:08:48.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.177013 829 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:08:48.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.177397 829 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:08:48.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.177629 829 server_base.cc:449] Attempting to create new FS layout instead 04:08:48.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.180986 829 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data/instance: 04:08:48.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "47032168a8194dd4acb45fe9a23758b0" 04:08:48.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:08:48 on dist-test-slave-dist-test-slave-fvsv" 04:08:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.181818 829 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal/instance: 04:08:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "47032168a8194dd4acb45fe9a23758b0" 04:08:48.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:08:48 on dist-test-slave-dist-test-slave-fvsv" 04:08:48.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.191586 829 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s 04:08:48.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.192858 829 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:08:48.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.193481 829 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:08:48.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.201129 829 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:08:48.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.201264 829 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal 04:08:48.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "47032168a8194dd4acb45fe9a23758b0" 04:08:48.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:08:48 on dist-test-slave-dist-test-slave-fvsv" 04:08:48.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:48.201601 829 fs_report.cc:352] FS layout report 04:08:48.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:08:48.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal 04:08:48.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal 04:08:48.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data/data 04:08:48.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:08:48.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:08:48.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:08:48.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:08:48.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:08:48.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:08:48.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:08:48.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:08:48.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:08:48.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:08:49.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.064374 829 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.829s user 0.002s sys 0.000s 04:08:49.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.064651 829 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.829s user 0.002s sys 0.000s 04:08:49.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.137259 829 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:08:49.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.201877 829 rpc_server.cc:205] RPC server started. Bound to: 127.0.202.126:46581 04:08:49.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.204324 829 webserver.cc:374] Webserver started at http://127.0.202.126:42729/ using document root and password file 04:08:49.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.205305 829 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data/info.pb 04:08:49.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.208649 891 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:08:49.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.211843 809 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 829 04:08:49.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.224833 891 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } has no permanent_uuid. Determining permanent_uuid... 04:08:49.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.0.202.125@KRBTEST.COM; defaulting to no policy 04:08:49.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:08:49.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.0.202.125@KRBTEST.COM" created. 04:08:49.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:49 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.126@KRBTEST.COM for kudu/127.0.202.126@KRBTEST.COM 04:08:49.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:08:49.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.0.202.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.125.keytab. 04:08:49.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.0.202.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.125.keytab. 04:08:49.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.0.202.125 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.125.keytab. 04:08:49.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.0.202.125 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.125.keytab. 04:08:49.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.258304 809 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:08:49.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:08:49.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal 04:08:49.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data 04:08:49.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:08:49.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:08:49.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:08:49.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:08:49.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.0.202.125:43873 04:08:49.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.0.202.125 04:08:49.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:08:49.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:08:49.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:08:49.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:08:49.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:08:49.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:08:49.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:08:49.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:08:49.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/logs 04:08:49.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data/info.pb 04:08:49.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:08:49.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:08:49.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:08:49.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:08:49.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.125.keytab 04:08:49.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.0.202.125 04:08:49.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:08:49.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:08:49.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:08:49.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:08:49.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:08:49.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.0.202.126:46581,127.0.202.125:43873,127.0.202.124:43241 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:08:49.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.264097 891 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } has no permanent_uuid. Determining permanent_uuid... 04:08:49.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.265807 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.125:43873: Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:08:49.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.285143 891 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } attempt: 1 04:08:49.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.286705 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.125:43873: Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:08:49.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.335005 891 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } attempt: 2 04:08:49.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.336710 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.125:43873: Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:08:49.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:08:49.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.406858 896 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:08:49.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.407204 896 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.0.202.125 04:08:49.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.407433 896 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:08:49.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.418817 896 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:08:49.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.418963 896 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:08:49.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.419090 896 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:08:49.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.419183 896 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:08:49.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.419292 896 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:08:49.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.419406 896 flags.cc:404] Enabled experimental flag: --principal=kudu/127.0.202.125 04:08:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.437871 896 master_main.cc:98] Master server non-default flags: 04:08:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data 04:08:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal 04:08:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:08:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:08:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.0.202.126:46581,127.0.202.125:43873,127.0.202.124:43241 04:08:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:08:49.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:08:49.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.0.202.125:43873 04:08:49.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:08:49.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:08:49.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.125.keytab 04:08:49.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.0.202.125 04:08:49.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:08:49.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:08:49.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:08:49.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:08:49.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:08:49.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data/info.pb 04:08:49.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.0.202.125 04:08:49.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:08:49.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:08:49.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:08:49.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:08:49.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:08:49.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:08:49.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/logs 04:08:49.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:08:49.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:08:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:08:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:08:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:08:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:08:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:08:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:08:49.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:08:49.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.447988 891 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } attempt: 3 04:08:49.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.449580 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.125:43873: Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:08:49.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.452785 896 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 101516us 04:08:49.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:49 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:08:49.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.455549 896 init.cc:359] Logged in from keytab as kudu/127.0.202.125@KRBTEST.COM (short username kudu) 04:08:49.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.459226 896 fs_manager.cc:263] Metadata directory not provided 04:08:49.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.459347 896 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:08:49.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.459574 896 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:08:49.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.459677 896 server_base.cc:449] Attempting to create new FS layout instead 04:08:49.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.462550 896 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data/instance: 04:08:49.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "391e3ec7324646ba844ec1e3e839f838" 04:08:49.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:08:49 on dist-test-slave-dist-test-slave-fvsv" 04:08:49.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.463281 896 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal/instance: 04:08:49.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "391e3ec7324646ba844ec1e3e839f838" 04:08:49.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:08:49 on dist-test-slave-dist-test-slave-fvsv" 04:08:49.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.472679 896 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.006s sys 0.003s 04:08:49.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.473839 896 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:08:49.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.474331 896 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:08:49.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.481173 896 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:08:49.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.481299 896 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal 04:08:49.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "391e3ec7324646ba844ec1e3e839f838" 04:08:49.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:08:49 on dist-test-slave-dist-test-slave-fvsv" 04:08:49.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.481508 896 fs_report.cc:352] FS layout report 04:08:49.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:08:49.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal 04:08:49.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal 04:08:49.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data/data 04:08:49.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:08:49.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:08:49.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:08:49.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:08:49.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:08:49.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:08:49.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:08:49.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:08:49.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:08:49.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:08:49.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.625932 891 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } attempt: 4 04:08:49.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.628254 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.125:43873: Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:08:49.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:49.931628 891 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } attempt: 5 04:08:49.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:49.933953 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.125:43873: Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:08:50.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.140311 896 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.625s user 0.002s sys 0.000s 04:08:50.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.140511 896 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.625s user 0.002s sys 0.000s 04:08:50.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.223423 896 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:08:50.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.289136 896 rpc_server.cc:205] RPC server started. Bound to: 127.0.202.125:43873 04:08:50.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.291644 896 webserver.cc:374] Webserver started at http://127.0.202.125:45079/ using document root and password file 04:08:50.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.292488 896 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data/info.pb 04:08:50.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.295575 809 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 896 04:08:50.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.295619 958 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:08:50.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.0.202.124@KRBTEST.COM; defaulting to no policy 04:08:50.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.314569 958 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } has no permanent_uuid. Determining permanent_uuid... 04:08:50.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:08:50.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.0.202.124@KRBTEST.COM" created. 04:08:50.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.125@KRBTEST.COM for kudu/127.0.202.126@KRBTEST.COM 04:08:50.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:08:50.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.0.202.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.124.keytab. 04:08:50.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.0.202.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.124.keytab. 04:08:50.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.0.202.124 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.124.keytab. 04:08:50.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.0.202.124 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.124.keytab. 04:08:50.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.346380 809 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:08:50.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:08:50.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/wal 04:08:50.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data 04:08:50.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.0.202.124:43241 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.0.202.124 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/logs 04:08:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data/info.pb 04:08:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:08:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:08:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:08:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:08:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.124.keytab 04:08:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.0.202.124 04:08:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:08:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:08:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:08:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:08:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:08:50.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.0.202.126:46581,127.0.202.125:43873,127.0.202.124:43241 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:08:50.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.348255 958 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } has no permanent_uuid. Determining permanent_uuid... 04:08:50.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.125@KRBTEST.COM for kudu/127.0.202.125@KRBTEST.COM 04:08:50.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.375633 958 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } has no permanent_uuid. Determining permanent_uuid... 04:08:50.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.377147 958 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:50.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.396517 958 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } attempt: 1 04:08:50.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.398099 958 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:50.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.446365 958 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } attempt: 2 04:08:50.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.448217 958 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:50.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.474182 891 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } attempt: 6 04:08:50.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.126@KRBTEST.COM for kudu/127.0.202.125@KRBTEST.COM 04:08:50.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.493485 891 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } has no permanent_uuid. Determining permanent_uuid... 04:08:50.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.495012 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:50.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:08:50.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.519101 964 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:08:50.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.519443 964 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.0.202.124 04:08:50.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.519670 964 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:08:50.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.527318 891 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } attempt: 1 04:08:50.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.528968 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:50.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.531666 964 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:08:50.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.531795 964 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:08:50.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.531903 964 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:08:50.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.532001 964 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:08:50.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.532120 964 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:08:50.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.532263 964 flags.cc:404] Enabled experimental flag: --principal=kudu/127.0.202.124 04:08:50.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.552812 964 master_main.cc:98] Master server non-default flags: 04:08:50.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data 04:08:50.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/wal 04:08:50.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:08:50.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:08:50.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.0.202.126:46581,127.0.202.125:43873,127.0.202.124:43241 04:08:50.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:08:50.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:08:50.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.0.202.124:43241 04:08:50.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:08:50.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:08:50.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.124.keytab 04:08:50.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.0.202.124 04:08:50.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:08:50.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:08:50.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:08:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:08:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:08:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data/info.pb 04:08:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.0.202.124 04:08:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:08:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:08:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:08:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:08:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:08:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:08:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/logs 04:08:50.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:08:50.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:08:50.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:08:50.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:08:50.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:08:50.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:08:50.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:08:50.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:08:50.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:08:50.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.559551 958 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } attempt: 3 04:08:50.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.561442 958 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:50.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.569958 964 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 102016us 04:08:50.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:08:50.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.572361 964 init.cc:359] Logged in from keytab as kudu/127.0.202.124@KRBTEST.COM (short username kudu) 04:08:50.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.575675 964 fs_manager.cc:263] Metadata directory not provided 04:08:50.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.575814 964 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:08:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.576164 964 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:08:50.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.576292 964 server_base.cc:449] Attempting to create new FS layout instead 04:08:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.579303 964 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data/instance: 04:08:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" 04:08:50.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:08:50 on dist-test-slave-dist-test-slave-fvsv" 04:08:50.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.580154 964 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/wal/instance: 04:08:50.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" 04:08:50.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:08:50 on dist-test-slave-dist-test-slave-fvsv" 04:08:50.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.589870 964 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.007s sys 0.002s 04:08:50.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.591241 964 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:08:50.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.591897 964 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:08:50.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.597256 891 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } attempt: 2 04:08:50.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.598816 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:50.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.600085 964 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s 04:08:50.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.600208 964 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/wal 04:08:50.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" 04:08:50.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:08:50 on dist-test-slave-dist-test-slave-fvsv" 04:08:50.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.600420 964 fs_report.cc:352] FS layout report 04:08:50.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:08:50.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/wal 04:08:50.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/wal 04:08:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data/data 04:08:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:08:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:08:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:08:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:08:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:08:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:08:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:08:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:08:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:08:50.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:08:50.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.675138 891 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } attempt: 3 04:08:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.676926 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:50.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.737768 958 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } attempt: 4 04:08:50.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.739536 958 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:50.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:50.814227 891 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } attempt: 4 04:08:50.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:50.815923 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:51.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.042836 958 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } attempt: 5 04:08:51.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:51.044780 958 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:51.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:51.026688 843 debug-util.cc:397] Leaking SignalData structure 0x7b080002a080 after lost signal to thread 830 04:08:51.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.099200 891 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } attempt: 5 04:08:51.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:51.101411 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:51.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:51.495844 964 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.869s user 0.357s sys 0.485s 04:08:51.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:51.496060 964 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.870s user 0.357s sys 0.485s 04:08:51.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.548280 964 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:08:51.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.579102 958 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } attempt: 6 04:08:51.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:51.584385 958 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.613735 891 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } attempt: 6 04:08:51.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:51.617542 891 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.0.202.124:43241: Network error: Client connection negotiation failed: client connection to 127.0.202.124:43241: connect: Connection refused (error 111) 04:08:51.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.675814 964 rpc_server.cc:205] RPC server started. Bound to: 127.0.202.124:43241 04:08:51.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.683825 964 webserver.cc:374] Webserver started at http://127.0.202.124:40001/ using document root and password file 04:08:51.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.684976 964 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data/info.pb 04:08:51.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.690076 1029 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:08:51.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.694509 809 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 964 04:08:51.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.717945 1029 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } has no permanent_uuid. Determining permanent_uuid... 04:08:51.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:51 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.124@KRBTEST.COM for kudu/127.0.202.126@KRBTEST.COM 04:08:51.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:51 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.124@KRBTEST.COM 04:08:51.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.762656 1029 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } has no permanent_uuid. Determining permanent_uuid... 04:08:51.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:51 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.124@KRBTEST.COM for kudu/127.0.202.125@KRBTEST.COM 04:08:51.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:51 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.125@KRBTEST.COM 04:08:51.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.799357 1029 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } has no permanent_uuid. Determining permanent_uuid... 04:08:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:51 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.124@KRBTEST.COM for kudu/127.0.202.124@KRBTEST.COM 04:08:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:51 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.126@KRBTEST.COM 04:08:51.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.820158 809 external_mini_cluster.cc:628] 0 TS(s) registered with all masters 04:08:51.820 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:08:51.821 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_kdc_env_vars { } 04:08:51.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.826042 1029 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: Bootstrap starting. 04:08:51.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.828862 1029 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: No blocks or log segments found. Creating new log. 04:08:51.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.831143 1029 log.cc:525] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: Log is configured to *not* fsync() on all Append() calls 04:08:51.829 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: get_kdc_env_vars { env_vars { key: "KRB5_CONFIG" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5.conf" } env_vars { key: "KRB5CCNAME" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5cc" } env_vars { key: "KUDU_ENABLE_KRB5_REALM_FIX" value: "yes" } env_vars { key: "KRB5_KDC_PROFILE" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kdc.conf" } } 04:08:51.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.835537 1029 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: No bootstrap required, opened a new log 04:08:51.835 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_masters { } 04:08:51.840 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.0.202.126" port: 46581 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.0.202.125" port: 43873 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.0.202.124" port: 43241 } } } 04:08:51.851 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_tservers { } 04:08:51.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.851294 1029 raft_consensus.cc:340] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } 04:08:51.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.851855 1029 raft_consensus.cc:366] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:08:51.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.852057 1029 raft_consensus.cc:705] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481fbe6d96fe4e60ba4e8ba309bf7fd7, State: Initialized, Role: FOLLOWER 04:08:51.852 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:08:51.852 [INFO - Time-limited test] (MiniKuduCluster.java:439) Running kinit for user test-admin 04:08:51.853 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: kinit { username: "test-admin" } 04:08:51.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.852890 1029 consensus_queue.cc:229] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } 04:08:51.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.856524 1037 sys_catalog.cc:339] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } } 04:08:51.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.857219 1037 sys_catalog.cc:342] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [sys.catalog]: This master's current role is: FOLLOWER 04:08:51.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.857666 1029 sys_catalog.cc:339] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } } 04:08:51.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.858247 1029 sys_catalog.cc:342] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [sys.catalog]: This master's current role is: FOLLOWER 04:08:51.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.859438 1029 sys_catalog.cc:427] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [sys.catalog]: configured and running, proceeding with master startup. 04:08:51.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:51.861999 1038 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:08:51.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:51 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:08:51.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:08:51.887 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:08:52.020 [DEBUG - Time-limited test] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5cc 04:08:52.040 [DEBUG - Time-limited test] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:08:52.088 [DEBUG - Time-limited test] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5cc 04:08:52.123 [DEBUG - Time-limited test] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:08:52.146 [DEBUG - Time-limited test] (SecurityContext.java:153) Using caller-provided subject with Kerberos principal test-admin@KRBTEST.COM. Caller is responsible for refreshing credentials. 04:08:52.148 [INFO - Time-limited test] (MiniKuduCluster.java:439) Running kinit for user test-admin 04:08:52.148 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: kinit { username: "test-admin" } 04:08:52.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:52.055522 983 tablet.cc:1799] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:08:52.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:52 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:08:52.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:08:52.182 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:08:52.182 [DEBUG - Time-limited test] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5cc 04:08:52.184 [DEBUG - Time-limited test] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:08:52.184 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.0.202.126:46581 04:08:52.186 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:08:52.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.187183 809 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 829 04:08:52.212 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:08:52.212 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.0.202.124:43241 04:08:52.212 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:08:52.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.213361 809 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 964 04:08:52.227 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:08:52.228 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.0.202.125:43873 04:08:52.228 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:08:52.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.229019 809 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 896 04:08:52.243 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:08:52.244 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.0.202.126:46581 04:08:52.245 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:08:52.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.247201 809 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:08:52.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:08:52.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal 04:08:52.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data 04:08:52.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:08:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:08:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:08:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:08:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.0.202.126:46581 04:08:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.0.202.126 04:08:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=42729 04:08:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:08:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:08:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:08:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:08:52.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/logs 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data/info.pb 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.126.keytab 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.0.202.126 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:08:52.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.0.202.126:46581,127.0.202.125:43873,127.0.202.124:43241 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:08:52.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:08:52.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:52.395095 1061 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:08:52.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:52.395436 1061 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.0.202.126 04:08:52.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:52.395738 1061 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:08:52.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:52.410532 1061 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:08:52.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:52.410676 1061 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:08:52.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:52.410784 1061 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:08:52.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:52.410873 1061 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:08:52.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:52.410987 1061 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:08:52.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:52.411135 1061 flags.cc:404] Enabled experimental flag: --principal=kudu/127.0.202.126 04:08:52.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.431607 1061 master_main.cc:98] Master server non-default flags: 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.0.202.126:46581,127.0.202.125:43873,127.0.202.124:43241 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.0.202.126:46581 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.126.keytab 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.0.202.126 04:08:52.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data/info.pb 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.0.202.126 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=42729 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/logs 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:08:52.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:08:52.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:08:52.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.445945 1061 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 103016us 04:08:52.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:52 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:08:52.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.448321 1061 init.cc:359] Logged in from keytab as kudu/127.0.202.126@KRBTEST.COM (short username kudu) 04:08:52.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.451095 1061 fs_manager.cc:263] Metadata directory not provided 04:08:52.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.451253 1061 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:08:52.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.456349 1061 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s 04:08:52.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.456992 1061 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:08:52.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.457559 1061 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:08:52.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.464278 1061 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:08:52.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.464414 1061 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal 04:08:52.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "47032168a8194dd4acb45fe9a23758b0" 04:08:52.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:08:48 on dist-test-slave-dist-test-slave-fvsv" 04:08:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:52.465742 1061 fs_report.cc:352] FS layout report 04:08:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:08:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal 04:08:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal 04:08:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data/data 04:08:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:08:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:08:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:08:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:08:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:08:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:08:52.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:08:52.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:08:52.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:08:52.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:08:53.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:53.097957 1061 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.608s user 0.256s sys 0.352s 04:08:53.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:53.098250 1061 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.608s user 0.256s sys 0.352s 04:08:53.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.125995 1061 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:08:53.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.198158 1061 rpc_server.cc:205] RPC server started. Bound to: 127.0.202.126:46581 04:08:53.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.201449 1061 webserver.cc:374] Webserver started at http://127.0.202.126:42729/ using document root and password file 04:08:53.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.202679 1061 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/data/info.pb 04:08:53.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.210346 1123 sys_catalog.cc:188] Verifying existing consensus state 04:08:53.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) E0627 04:08:53.210952 1123 master.cc:205] Unable to init master catalog manager: Not found: Unable to initialize catalog manager: Failed to initialize sys tables async: Unable to load consensus metadata for tablet 00000000000000000000000000000000: Unable to load consensus metadata for tablet 00000000000000000000000000000000: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal/consensus-meta/00000000000000000000000000000000: No such file or directory (error 2) 04:08:53.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) F0627 04:08:53.211258 1061 master_main.cc:107] Check failed: _s.ok() Bad status: Not found: Unable to initialize catalog manager: Failed to initialize sys tables async: Unable to load consensus metadata for tablet 00000000000000000000000000000000: Unable to load consensus metadata for tablet 00000000000000000000000000000000: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-0/wal/consensus-meta/00000000000000000000000000000000: No such file or directory (error 2) 04:08:53.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) *** Check failure stack trace: *** 04:08:53.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.211822 809 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 1061 04:08:53.212 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:08:53.213 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.0.202.124:43241 04:08:53.213 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:08:53.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.214809 809 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:08:53.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:08:53.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/wal 04:08:53.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data 04:08:53.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:08:53.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:08:53.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:08:53.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:08:53.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.0.202.124:43241 04:08:53.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.0.202.124 04:08:53.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=40001 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/logs 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data/info.pb 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:08:53.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.124.keytab 04:08:53.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.0.202.124 04:08:53.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:08:53.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:08:53.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:08:53.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:08:53.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:08:53.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.0.202.126:46581,127.0.202.125:43873,127.0.202.124:43241 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:08:53.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) @ 0x7f63bd5a63c2 google::LogMessage::Flush() 04:08:53.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) @ 0x7f63bd5aabdb google::LogMessageFatal::~LogMessageFatal() 04:08:53.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) @ 0x4cc2e8 kudu::master::MasterMain() 04:08:53.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) @ 0x4cbd1f main 04:08:53.250 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) @ 0x7f63b9e7cf45 __libc_start_main 04:08:53.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) @ 0x4224f9 (unknown) 04:08:53.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:08:53.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:53.373611 1124 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:08:53.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:53.373927 1124 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.0.202.124 04:08:53.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:53.374166 1124 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:08:53.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:53.384951 1124 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:08:53.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:53.385090 1124 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:08:53.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:53.385197 1124 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:08:53.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:53.385284 1124 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:08:53.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:53.385418 1124 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:08:53.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:53.385577 1124 flags.cc:404] Enabled experimental flag: --principal=kudu/127.0.202.124 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.404785 1124 master_main.cc:98] Master server non-default flags: 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/wal 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.0.202.126:46581,127.0.202.125:43873,127.0.202.124:43241 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.0.202.124:43241 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.124.keytab 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.0.202.124 04:08:53.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data/info.pb 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.0.202.124 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=40001 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/logs 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:08:53.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:08:53.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.420635 1124 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 103516us 04:08:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:53 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:08:53.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.422797 1124 init.cc:359] Logged in from keytab as kudu/127.0.202.124@KRBTEST.COM (short username kudu) 04:08:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.425468 1124 fs_manager.cc:263] Metadata directory not provided 04:08:53.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.425601 1124 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:08:53.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.430501 1124 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s 04:08:53.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.431140 1124 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:08:53.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.431629 1124 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:08:53.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.439220 1124 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:08:53.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.439369 1124 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/wal 04:08:53.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" 04:08:53.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:08:50 on dist-test-slave-dist-test-slave-fvsv" 04:08:53.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:53.440627 1124 fs_report.cc:352] FS layout report 04:08:53.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/wal 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/wal 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data/data 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:08:53.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:08:54.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.262853 1124 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.798s user 0.297s sys 0.500s 04:08:54.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.263074 1124 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.798s user 0.297s sys 0.500s 04:08:54.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.328991 1124 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:08:54.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.401235 1124 rpc_server.cc:205] RPC server started. Bound to: 127.0.202.124:43241 04:08:54.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.403748 1124 webserver.cc:374] Webserver started at http://127.0.202.124:40001/ using document root and password file 04:08:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.404688 1124 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-2/data/info.pb 04:08:54.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.412037 1186 sys_catalog.cc:188] Verifying existing consensus state 04:08:54.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.412717 809 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 1124 04:08:54.413 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:08:54.413 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.0.202.125:43873 04:08:54.413 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:08:54.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.414736 809 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.0.202.125:43873 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.0.202.125 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45079 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:08:54.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/logs 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data/info.pb 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.125.keytab 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.0.202.125 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:08:54.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.0.202.126:46581,127.0.202.125:43873,127.0.202.124:43241 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:08:54.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.416317 1186 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: Bootstrap starting. 04:08:54.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.442528 1186 log.cc:525] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: Log is configured to *not* fsync() on all Append() calls 04:08:54.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.448907 1186 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:08:54.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.449438 1186 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: Bootstrap complete. 04:08:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.469137 1186 raft_consensus.cc:340] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } 04:08:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.474198 1186 raft_consensus.cc:366] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:08:54.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.474354 1186 raft_consensus.cc:705] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481fbe6d96fe4e60ba4e8ba309bf7fd7, State: Initialized, Role: FOLLOWER 04:08:54.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.474895 1186 consensus_queue.cc:229] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } 04:08:54.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.476948 1186 sys_catalog.cc:339] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } } 04:08:54.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.477427 1186 sys_catalog.cc:342] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [sys.catalog]: This master's current role is: FOLLOWER 04:08:54.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.478564 1186 sys_catalog.cc:427] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [sys.catalog]: configured and running, proceeding with master startup. 04:08:54.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.477371 1189 sys_catalog.cc:339] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } } 04:08:54.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.546480 1189 sys_catalog.cc:342] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [sys.catalog]: This master's current role is: FOLLOWER 04:08:54.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.553480 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:08:54.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.553776 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:08:54.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.593264 1142 tablet.cc:1799] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:08:54.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:08:54.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.595535 1187 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:08:54.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.595872 1187 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.0.202.125 04:08:54.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.596221 1187 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:08:54.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.607331 1187 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:08:54.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.607484 1187 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:08:54.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.607635 1187 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:08:54.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.607764 1187 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:08:54.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.607939 1187 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:08:54.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.608144 1187 flags.cc:404] Enabled experimental flag: --principal=kudu/127.0.202.125 04:08:54.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.627122 1187 master_main.cc:98] Master server non-default flags: 04:08:54.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data 04:08:54.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal 04:08:54.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:08:54.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:08:54.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.0.202.126:46581,127.0.202.125:43873,127.0.202.124:43241 04:08:54.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.0.202.125:43873 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/krb5kdc/kudu_127.0.202.125.keytab 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.0.202.125 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data/info.pb 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.0.202.125 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45079 04:08:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/logs 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:08:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:08:54.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.644246 1187 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 104016us 04:08:54.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:54 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:08:54.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.646853 1187 init.cc:359] Logged in from keytab as kudu/127.0.202.125@KRBTEST.COM (short username kudu) 04:08:54.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.650679 1187 fs_manager.cc:263] Metadata directory not provided 04:08:54.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.650852 1187 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:08:54.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.656322 1187 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s 04:08:54.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.656953 1187 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:08:54.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.657512 1187 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:08:54.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.664424 1187 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s 04:08:54.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.664559 1187 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "391e3ec7324646ba844ec1e3e839f838" 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:08:49 on dist-test-slave-dist-test-slave-fvsv" 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.665902 1187 fs_report.cc:352] FS layout report 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data/data 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:08:54.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:08:54.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:08:54.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:08:54.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:08:54.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.835070 1189 raft_consensus.cc:472] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:08:54.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.836205 1189 raft_consensus.cc:494] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } 04:08:54.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.844718 1189 leader_election.cc:296] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581), 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873) 04:08:54.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.849172 1133 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581): Network error: Client connection negotiation failed: client connection to 127.0.202.126:46581: connect: Connection refused (error 111) 04:08:54.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:54.853420 1137 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873): Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:08:54.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.853782 1137 leader_election.cc:310] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 481fbe6d96fe4e60ba4e8ba309bf7fd7; no voters: 391e3ec7324646ba844ec1e3e839f838, 47032168a8194dd4acb45fe9a23758b0 04:08:54.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:54.855618 1189 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority 04:08:55.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:55.326938 1187 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.628s user 0.001s sys 0.001s 04:08:55.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:55.327210 1187 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.629s user 0.002s sys 0.001s 04:08:55.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:55.386312 1187 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:08:55.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:55.543624 1187 rpc_server.cc:205] RPC server started. Bound to: 127.0.202.125:43873 04:08:55.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:55.547511 1187 webserver.cc:374] Webserver started at http://127.0.202.125:45079/ using document root and password file 04:08:55.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:55.548462 1187 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/data/info.pb 04:08:55.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:55.551936 809 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 1187 04:08:55.552 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:08:55.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:55.556169 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:08:55.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:55.556404 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:08:55.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:55.560945 1255 sys_catalog.cc:188] Verifying existing consensus state 04:08:55.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) E0627 04:08:55.561691 1255 master.cc:205] Unable to init master catalog manager: Not found: Unable to initialize catalog manager: Failed to initialize sys tables async: Unable to load consensus metadata for tablet 00000000000000000000000000000000: Unable to load consensus metadata for tablet 00000000000000000000000000000000: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal/consensus-meta/00000000000000000000000000000000: No such file or directory (error 2) 04:08:55.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) F0627 04:08:55.561955 1187 master_main.cc:107] Check failed: _s.ok() Bad status: Not found: Unable to initialize catalog manager: Failed to initialize sys tables async: Unable to load consensus metadata for tablet 00000000000000000000000000000000: Unable to load consensus metadata for tablet 00000000000000000000000000000000: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6042585889828166457/master-1/wal/consensus-meta/00000000000000000000000000000000: No such file or directory (error 2) 04:08:55.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) *** Check failure stack trace: *** 04:08:55.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) @ 0x7fc12ce673c2 google::LogMessage::Flush() 04:08:55.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) @ 0x7fc12ce6bbdb google::LogMessageFatal::~LogMessageFatal() 04:08:55.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) @ 0x4cc2e8 kudu::master::MasterMain() 04:08:55.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) @ 0x4cbd1f main 04:08:55.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) @ 0x7fc12973df45 __libc_start_main 04:08:55.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) @ 0x4224f9 (unknown) 04:08:55.630 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:55.631 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:55.632 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:55.684 [WARN - New I/O worker #11] (SecurityContext.java:201) Caller-provided Subject has a Kerberos ticket that is about to expire. Kudu expects the application to renew or re-acquire its own tickets before expiration. 04:08:55.695 [DEBUG - New I/O worker #11] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.0.202.124 04:08:55.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:08:55 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.202.124@KRBTEST.COM 04:08:55.794 [ERROR - New I/O worker #12] (Connection.java:461) [peer master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception from downstream on [id: 0x2fae7299, /127.0.0.1:57014 => /127.0.202.125:43873] java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_141] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_141] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_141] at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.8.0_141] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_141] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:?] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:?] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141] 04:08:55.813 [DEBUG - New I/O worker #12] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state NEGOTIATING due to: [peer master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception from downstream on [id: 0x2fae7299, /127.0.0.1:57014 => /127.0.202.125:43873] 04:08:55.814 [INFO - New I/O worker #12] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: [peer master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception from downstream on [id: 0x2fae7299, /127.0.0.1:57014 => /127.0.202.125:43873] 04:08:55.836 [DEBUG - New I/O worker #11] (Negotiator.java:764) Authenticated connection [id: 0x1d2a2502, /127.0.0.1:57604 => /127.0.202.124:43241] using SASL/GSSAPI 04:08:55.898 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:55.922 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:55.922 [DEBUG - New I/O worker #13] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:55.923 [INFO - New I/O worker #13] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:55.924 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:55.924 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:55.924 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:55.926 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:55.942 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:55.942 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:55.943 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:55.944 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:55.944 [DEBUG - New I/O worker #16] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:55.946 [INFO - New I/O worker #16] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:55.946 [WARN - New I/O worker #16] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:55.962 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:55.962 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:55.971 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:55.973 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:55.973 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:55.973 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:55.974 [WARN - New I/O boss #18] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:55.980 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:55.981 [DEBUG - New I/O worker #11] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:55.981 [INFO - New I/O worker #11] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:55.982 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:55.982 [DEBUG - New I/O worker #12] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:55.983 [INFO - New I/O worker #12] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:55.985 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:56.021 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:56.022 [DEBUG - New I/O worker #13] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:56.024 [INFO - New I/O worker #13] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:56.025 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:56.025 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:56.027 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:56.027 [WARN - New I/O boss #18] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:56.083 [DEBUG - New I/O worker #15] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:56.084 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:56.084 [INFO - New I/O worker #15] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:56.089 [DEBUG - New I/O worker #16] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:56.089 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:56.089 [INFO - New I/O worker #16] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:56.090 [WARN - New I/O worker #16] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:56.162 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:56.164 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:56.164 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:56.165 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:56.166 [DEBUG - New I/O worker #10] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:56.167 [INFO - New I/O worker #10] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:56.168 [WARN - New I/O worker #10] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:56.301 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:56.301 [DEBUG - New I/O worker #11] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:56.302 [INFO - New I/O worker #11] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:56.303 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:56.304 [DEBUG - New I/O worker #12] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:56.304 [INFO - New I/O worker #12] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:56.307 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:56.363 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:56.363 [DEBUG - New I/O worker #13] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:56.364 [INFO - New I/O worker #13] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:56.365 [DEBUG - New I/O worker #14] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:56.365 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:56.366 [INFO - New I/O worker #14] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:56.367 [WARN - New I/O worker #14] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:56.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:56.558254 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:08:56.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:56.558455 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:08:56.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:56.756907 1258 raft_consensus.cc:472] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:08:56.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:56.757220 1258 raft_consensus.cc:494] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } 04:08:56.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:56.758296 1258 leader_election.cc:296] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581), 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873) 04:08:56.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:56.760970 1133 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581): Network error: Client connection negotiation failed: client connection to 127.0.202.126:46581: connect: Connection refused (error 111) 04:08:56.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:56.761214 1137 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873): Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:08:56.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:56.761430 1137 leader_election.cc:310] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 481fbe6d96fe4e60ba4e8ba309bf7fd7; no voters: 391e3ec7324646ba844ec1e3e839f838, 47032168a8194dd4acb45fe9a23758b0 04:08:56.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:56.761911 1258 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority 04:08:56.984 [DEBUG - New I/O worker #15] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:56.984 [INFO - New I/O worker #15] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:56.984 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:56.985 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:56.986 [DEBUG - New I/O worker #16] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:56.986 [INFO - New I/O worker #16] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:56.986 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:57.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:57.560364 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:08:57.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:57.560562 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:08:58.102 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:58.103 [DEBUG - New I/O worker #17] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:58.104 [INFO - New I/O worker #17] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:58.105 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:58.105 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:58.105 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:58.106 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:58.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:58.562331 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:08:58.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:58.562570 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:08:59.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:59.173174 1262 raft_consensus.cc:472] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:08:59.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:59.173481 1262 raft_consensus.cc:494] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } 04:08:59.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:59.174553 1262 leader_election.cc:296] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581), 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873) 04:08:59.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:59.181386 1133 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581): Network error: Client connection negotiation failed: client connection to 127.0.202.126:46581: connect: Connection refused (error 111) 04:08:59.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:59.182276 1137 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873): Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:08:59.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:59.182464 1137 leader_election.cc:310] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 481fbe6d96fe4e60ba4e8ba309bf7fd7; no voters: 391e3ec7324646ba844ec1e3e839f838, 47032168a8194dd4acb45fe9a23758b0 04:08:59.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:08:59.182938 1262 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority 04:08:59.422 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:08:59.424 [DEBUG - New I/O worker #11] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:59.424 [INFO - New I/O worker #11] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:08:59.428 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:08:59.428 [DEBUG - New I/O worker #12] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:59.429 [INFO - New I/O worker #12] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:08:59.429 [WARN - New I/O worker #12] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:08:59.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:59.564721 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:08:59.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:08:59.564903 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:00.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:00.566769 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:00.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:00.566956 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:00.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:00.787484 1265 raft_consensus.cc:472] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:09:00.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:00.787793 1265 raft_consensus.cc:494] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } 04:09:00.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:00.788836 1265 leader_election.cc:296] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581), 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873) 04:09:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:00.792174 1133 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581): Network error: Client connection negotiation failed: client connection to 127.0.202.126:46581: connect: Connection refused (error 111) 04:09:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:00.792477 1137 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873): Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:09:00.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:00.792704 1137 leader_election.cc:310] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 481fbe6d96fe4e60ba4e8ba309bf7fd7; no voters: 391e3ec7324646ba844ec1e3e839f838, 47032168a8194dd4acb45fe9a23758b0 04:09:00.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:00.793177 1265 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority 04:09:01.301 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:01.302 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:01.302 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:01.303 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:01.303 [DEBUG - New I/O worker #14] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:01.304 [INFO - New I/O worker #14] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:01.304 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:01.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:01.568744 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:01.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:01.568950 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:02.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:02.570684 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:02.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:02.570883 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:03.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:03 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:03.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:03.430423 1130 init.cc:303] Successfully reacquired a new kerberos TGT 04:09:03.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:03.573379 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:03.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:03.573598 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:04.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:04.575357 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:04.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:04.575543 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:04.821 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:04.822 [DEBUG - New I/O worker #15] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:04.822 [INFO - New I/O worker #15] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:04.823 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:04.823 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:04.823 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:04.824 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:05.482 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:05.483 [DEBUG - New I/O worker #17] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:05.483 [INFO - New I/O worker #17] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:05.484 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:05.484 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:05.484 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:05.486 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:05.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:05.577296 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:05.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:05.577483 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:06.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:06.579283 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:06.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:06.579483 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:06.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:06.644618 1269 raft_consensus.cc:472] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:09:06.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:06.644994 1269 raft_consensus.cc:494] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } 04:09:06.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:06.646220 1269 leader_election.cc:296] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581), 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873) 04:09:06.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:06.649888 1133 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581): Network error: Client connection negotiation failed: client connection to 127.0.202.126:46581: connect: Connection refused (error 111) 04:09:06.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:06.650285 1137 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873): Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:09:06.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:06.650480 1137 leader_election.cc:310] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 481fbe6d96fe4e60ba4e8ba309bf7fd7; no voters: 391e3ec7324646ba844ec1e3e839f838, 47032168a8194dd4acb45fe9a23758b0 04:09:06.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:06.650948 1269 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority 04:09:07.402 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:07.403 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:07.403 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:07.404 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:07.404 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:07.405 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:07.406 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:07.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:07.581202 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:07.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:07.581403 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:08.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:08.583338 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:08.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:08.583536 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:08.642 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:08.642 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:08.643 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:08.644 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:08.644 [DEBUG - New I/O worker #14] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:08.644 [INFO - New I/O worker #14] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:08.645 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:09.585330 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:09.585528 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:10.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:10.587427 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:10.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:10.587617 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:11.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:11.589398 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:11.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:11.589599 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:11.642 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:11.642 [DEBUG - New I/O worker #15] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:11.642 [INFO - New I/O worker #15] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:11.643 [DEBUG - New I/O worker #16] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:11.644 [INFO - New I/O worker #16] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:11.643 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:11.646 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:12.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:12.465893 1273 raft_consensus.cc:472] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:09:12.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:12.466187 1273 raft_consensus.cc:494] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } 04:09:12.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:12.467447 1273 leader_election.cc:296] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581), 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873) 04:09:12.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:12.472363 1133 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581): Network error: Client connection negotiation failed: client connection to 127.0.202.126:46581: connect: Connection refused (error 111) 04:09:12.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:12.472637 1137 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873): Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:09:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:12.472863 1137 leader_election.cc:310] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 481fbe6d96fe4e60ba4e8ba309bf7fd7; no voters: 391e3ec7324646ba844ec1e3e839f838, 47032168a8194dd4acb45fe9a23758b0 04:09:12.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:12.473429 1273 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority 04:09:12.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:12.591420 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:12.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:12.591626 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:13.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:13 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:13.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:13.432199 1130 init.cc:303] Successfully reacquired a new kerberos TGT 04:09:13.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:13.594027 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:13.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:13.594280 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:13.881 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:13.881 [DEBUG - New I/O worker #17] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:13.881 [INFO - New I/O worker #17] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:13.882 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:13.883 [DEBUG - New I/O worker #10] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:13.883 [INFO - New I/O worker #10] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:13.884 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:14.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:14.596107 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:14.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:14.596299 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:15.020 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:15.021 [DEBUG - New I/O worker #11] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:15.021 [INFO - New I/O worker #11] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:15.021 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:15.021 [DEBUG - New I/O worker #12] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:15.021 [INFO - New I/O worker #12] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:15.024 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:15.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:15.598091 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:15.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:15.598294 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:16.021 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:16.021 [DEBUG - New I/O worker #13] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:16.021 [INFO - New I/O worker #13] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:16.022 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:16.022 [DEBUG - New I/O worker #14] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:16.022 [INFO - New I/O worker #14] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:16.024 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:16.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:16.600638 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:16.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:16.600896 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:17.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:17.602684 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:17.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:17.602874 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:18.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:18.192281 1278 raft_consensus.cc:472] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:09:18.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:18.193845 1278 raft_consensus.cc:494] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47032168a8194dd4acb45fe9a23758b0" member_type: VOTER last_known_addr { host: "127.0.202.126" port: 46581 } } peers { permanent_uuid: "391e3ec7324646ba844ec1e3e839f838" member_type: VOTER last_known_addr { host: "127.0.202.125" port: 43873 } } peers { permanent_uuid: "481fbe6d96fe4e60ba4e8ba309bf7fd7" member_type: VOTER last_known_addr { host: "127.0.202.124" port: 43241 } } 04:09:18.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:18.195196 1278 leader_election.cc:296] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581), 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873) 04:09:18.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:18.197276 1133 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 47032168a8194dd4acb45fe9a23758b0 (127.0.202.126:46581): Network error: Client connection negotiation failed: client connection to 127.0.202.126:46581: connect: Connection refused (error 111) 04:09:18.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:18.198520 1137 leader_election.cc:341] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 391e3ec7324646ba844ec1e3e839f838 (127.0.202.125:43873): Network error: Client connection negotiation failed: client connection to 127.0.202.125:43873: connect: Connection refused (error 111) 04:09:18.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:18.198791 1137 leader_election.cc:310] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 481fbe6d96fe4e60ba4e8ba309bf7fd7; no voters: 391e3ec7324646ba844ec1e3e839f838, 47032168a8194dd4acb45fe9a23758b0 04:09:18.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:18.199434 1278 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority 04:09:18.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:18.604635 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:18.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:18.604820 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:19.041 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:19.041 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:19.042 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:19.042 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:19.042 [DEBUG - New I/O worker #16] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:19.042 [INFO - New I/O worker #16] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:19.044 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:19.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:19.099160 1138 debug-util.cc:397] Leaking SignalData structure 0x7b080002ae20 after lost signal to thread 1125 04:09:19.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:19.099743 1138 debug-util.cc:397] Leaking SignalData structure 0x7b080002a2e0 after lost signal to thread 1184 04:09:19.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:19.606521 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:19.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:19.606722 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:20.481 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:20.481 [DEBUG - New I/O worker #17] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:20.482 [INFO - New I/O worker #17] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:20.482 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:20.482 [DEBUG - New I/O worker #10] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:20.482 [INFO - New I/O worker #10] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:20.485 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:20.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:20.608510 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:20.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:20.608702 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:21.561 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:21.561 [DEBUG - New I/O worker #11] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:21.561 [INFO - New I/O worker #11] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:21.561 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:21.561 [DEBUG - New I/O boss #18] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:21.562 [INFO - New I/O boss #18] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:21.563 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:21.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:21.610440 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:21.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:21.610635 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:22.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:22.612381 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:22.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:22.612570 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:23.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:23 dist-test-slave-dist-test-slave-fvsv krb5kdc[818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608563, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.202.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:23.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:23.437232 1130 init.cc:303] Successfully reacquired a new kerberos TGT 04:09:23.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:23.615546 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:23.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:23.615820 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:24.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:24.617877 1190 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 481fbe6d96fe4e60ba4e8ba309bf7fd7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:24.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:24.618083 1190 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:25.161 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581 04:09:25.161 [DEBUG - New I/O worker #13] (Connection.java:691) [peer master-127.0.202.126:46581(127.0.202.126:46581)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:25.161 [INFO - New I/O worker #13] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.126:46581: java.net.ConnectException: Connection refused: /127.0.202.126:46581 04:09:25.162 [INFO - New I/O boss #18] (Connection.java:444) Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873 04:09:25.162 [DEBUG - New I/O worker #14] (Connection.java:691) [peer master-127.0.202.125:43873(127.0.202.125:43873)] cleaning up while in state CONNECTING due to: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:25.162 [INFO - New I/O worker #14] (ConnectToCluster.java:361) Unable to connect to master 127.0.202.125:43873: java.net.ConnectException: Connection refused: /127.0.202.125:43873 04:09:25.164 [WARN - New I/O worker #11] (ConnectToCluster.java:281) Unable to find the leader master 127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873; will retry 04:09:25.167 [DEBUG - New I/O worker #11] (AsyncKuduClient.java:1674) Cannot continue with RPC because of: Timed out: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=26, TimeoutTracker(timeout=30000, elapsed=29608), Traces: [0ms] refreshing cache from master, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [63ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [69ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581, [70ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [250ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: [peer master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception from downstream on [id: 0x2fae7299, /127.0.0.1:57014 => /127.0.202.125:43873], [282ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [336ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: [peer master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception from downstream on [id: 0x2fae7299, /127.0.0.1:57014 => /127.0.202.125:43873], [357ms] refreshing cache from master, [358ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [358ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [360ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [360ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [361ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [363ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [364ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [376ms] refreshing cache from master, [377ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [377ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [378ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [379ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581, [381ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [382ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [383ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [397ms] refreshing cache from master, [397ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [398ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [399ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581, [402ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [408ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [410ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [411ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [416ms] refreshing cache from master, [416ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [417ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [418ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [418ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [419ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [421ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [422ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [457ms] refreshing cache from master, [457ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [457ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [460ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [460ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [462ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [463ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [465ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [517ms] refreshing cache from master, [517ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [517ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [518ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [520ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [523ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [526ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [527ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [597ms] refreshing cache from master, [597ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [598ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [599ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [601ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581, [602ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [604ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [605ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [736ms] refreshing cache from master, [737ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [737ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [738ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [739ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [741ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [743ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [744ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [797ms] refreshing cache from master, [797ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [798ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [798ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [801ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [802ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [802ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [804ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [1417ms] refreshing cache from master, [1417ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [1417ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [1421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [1421ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [1423ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [1423ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [1424ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [2537ms] refreshing cache from master, [2537ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [2537ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [2540ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [2541ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [2542ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [2543ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [2543ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [3857ms] refreshing cache from master, [3857ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [3858ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [3861ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [3861ms] too many traces: truncated at 100 traces, deferred=Deferred@531938157(state=PENDING, result=null, callback=(continuation of Deferred@691636556 after retry RPC after error@610503197) -> (continuation of Deferred@1438562118 after retry RPC after error@1877676367) -> (continuation of Deferred@176002583 after retry RPC after error@855033383) -> (continuation of Deferred@315773880 after retry RPC after error@621038020) -> (continuation of Deferred@1127198301 after retry RPC after error@1206685665) -> (continuation of Deferred@1201883959 after retry RPC after error@1089368174) -> (continuation of Deferred@2114810728 after retry RPC after error@1564305615) -> (continuation of Deferred@1735340 after retry RPC after error@868489153) -> (continuation of Deferred@642229474 after retry RPC after error@269147125) -> (continuation of Deferred@1125522498 after retry RPC after error@1358918142) -> (continuation of Deferred@1258602423 after retry RPC after error@219615313) -> (continuation of Deferred@2115041725 after retry RPC after error@1430980352) -> (continuation of Deferred@1004335897 after retry RPC after error@243071403) -> (continuation of Deferred@2002269384 after retry RPC after error@2049334820) -> (continuation of Deferred@650001986 after retry RPC after error@1340484255) -> (continuation of Deferred@2033105375 after retry RPC after error@825298848) -> (continuation of Deferred@1722444778 after retry RPC after error@417732466) -> (continuation of Deferred@344634864 after retry RPC after error@763356366) -> (continuation of Deferred@406794160 after retry RPC after error@810214978) -> (continuation of Deferred@1907286158 after retry RPC after error@589636577) -> (continuation of Deferred@1107745434 after retry RPC after error@630803566) -> (continuation of Deferred@590705347 after retry RPC after error@696608623) -> (continuation of Deferred@1571007183 after retry RPC after error@850814761) -> (continuation of Deferred@983414998 after retry RPC after error@950083317) -> (continuation of Deferred@2014588668 after retry RPC after error@2040173048), errback=(continuation of Deferred@691636556 after retry RPC after error@610503197) -> (continuation of Deferred@1438562118 after retry RPC after error@1877676367) -> (continuation of Deferred@176002583 after retry RPC after error@855033383) -> (continuation of Deferred@315773880 after retry RPC after error@621038020) -> (continuation of Deferred@1127198301 after retry RPC after error@1206685665) -> (continuation of Deferred@1201883959 after retry RPC after error@1089368174) -> (continuation of Deferred@2114810728 after retry RPC after error@1564305615) -> (continuation of Deferred@1735340 after retry RPC after error@868489153) -> (continuation of Deferred@642229474 after retry RPC after error@269147125) -> (continuation of Deferred@1125522498 after retry RPC after error@1358918142) -> (continuation of Deferred@1258602423 after retry RPC after error@219615313) -> (continuation of Deferred@2115041725 after retry RPC after error@1430980352) -> (continuation of Deferred@1004335897 after retry RPC after error@243071403) -> (continuation of Deferred@2002269384 after retry RPC after error@2049334820) -> (continuation of Deferred@650001986 after retry RPC after error@1340484255) -> (continuation of Deferred@2033105375 after retry RPC after error@825298848) -> (continuation of Deferred@1722444778 after retry RPC after error@417732466) -> (continuation of Deferred@344634864 after retry RPC after error@763356366) -> (continuation of Deferred@406794160 after retry RPC after error@810214978) -> (continuation of Deferred@1907286158 after retry RPC after error@589636577) -> (continuation of Deferred@1107745434 after retry RPC after error@630803566) -> (continuation of Deferred@590705347 after retry RPC after error@696608623) -> (continuation of Deferred@1571007183 after retry RPC after error@850814761) -> (continuation of Deferred@983414998 after retry RPC after error@950083317) -> (continuation of Deferred@2014588668 after retry RPC after error@2040173048))) 04:09:25.172 [DEBUG - New I/O worker #11] (Connection.java:691) [peer master-127.0.202.124:43241(127.0.202.124:43241)] cleaning up while in state READY due to: connection disconnected 04:09:25.174 [DEBUG - Time-limited test] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:09:25.176 [DEBUG - main] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:09:25.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:25.177498 809 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 1124 04:09:26.216 [ERROR - main] (RetryRule.java:217) org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally: failed attempt 1 org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=26, TimeoutTracker(timeout=30000, elapsed=29608), Traces: [0ms] refreshing cache from master, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [63ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [69ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581, [70ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [250ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: [peer master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception from downstream on [id: 0x2fae7299, /127.0.0.1:57014 => /127.0.202.125:43873], [282ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [336ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: [peer master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception from downstream on [id: 0x2fae7299, /127.0.0.1:57014 => /127.0.202.125:43873], [357ms] refreshing cache from master, [358ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [358ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [360ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [360ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [361ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [363ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [364ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [376ms] refreshing cache from master, [377ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [377ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [378ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [379ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581, [381ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [382ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [383ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [397ms] refreshing cache from master, [397ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [398ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [399ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581, [402ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [408ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [410ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [411ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [416ms] refreshing cache from master, [416ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [417ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [418ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [418ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [419ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [421ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [422ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [457ms] refreshing cache from master, [457ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [457ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [460ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [460ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [462ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [463ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [465ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [517ms] refreshing cache from master, [517ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [517ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [518ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [520ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [523ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [526ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [527ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [597ms] refreshing cache from master, [597ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [598ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [599ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [601ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581, [602ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [604ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [605ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [736ms] refreshing cache from master, [737ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [737ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [738ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [739ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [741ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [743ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [744ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [797ms] refreshing cache from master, [797ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [798ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [798ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [801ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [802ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [802ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [804ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [1417ms] refreshing cache from master, [1417ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [1417ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [1421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [1421ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [1423ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [1423ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [1424ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [2537ms] refreshing cache from master, [2537ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [2537ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [2540ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [2541ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [2542ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [2543ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [2543ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [3857ms] refreshing cache from master, [3857ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [3858ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [3861ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [3861ms] too many traces: truncated at 100 traces, deferred=Deferred@531938157(state=PENDING, result=null, callback=(continuation of Deferred@691636556 after retry RPC after error@610503197) -> (continuation of Deferred@1438562118 after retry RPC after error@1877676367) -> (continuation of Deferred@176002583 after retry RPC after error@855033383) -> (continuation of Deferred@315773880 after retry RPC after error@621038020) -> (continuation of Deferred@1127198301 after retry RPC after error@1206685665) -> (continuation of Deferred@1201883959 after retry RPC after error@1089368174) -> (continuation of Deferred@2114810728 after retry RPC after error@1564305615) -> (continuation of Deferred@1735340 after retry RPC after error@868489153) -> (continuation of Deferred@642229474 after retry RPC after error@269147125) -> (continuation of Deferred@1125522498 after retry RPC after error@1358918142) -> (continuation of Deferred@1258602423 after retry RPC after error@219615313) -> (continuation of Deferred@2115041725 after retry RPC after error@1430980352) -> (continuation of Deferred@1004335897 after retry RPC after error@243071403) -> (continuation of Deferred@2002269384 after retry RPC after error@2049334820) -> (continuation of Deferred@650001986 after retry RPC after error@1340484255) -> (continuation of Deferred@2033105375 after retry RPC after error@825298848) -> (continuation of Deferred@1722444778 after retry RPC after error@417732466) -> (continuation of Deferred@344634864 after retry RPC after error@763356366) -> (continuation of Deferred@406794160 after retry RPC after error@810214978) -> (continuation of Deferred@1907286158 after retry RPC after error@589636577) -> (continuation of Deferred@1107745434 after retry RPC after error@630803566) -> (continuation of Deferred@590705347 after retry RPC after error@696608623) -> (continuation of Deferred@1571007183 after retry RPC after error@850814761) -> (continuation of Deferred@983414998 after retry RPC after error@950083317) -> (continuation of Deferred@2014588668 after retry RPC after error@2040173048), errback=(continuation of Deferred@691636556 after retry RPC after error@610503197) -> (continuation of Deferred@1438562118 after retry RPC after error@1877676367) -> (continuation of Deferred@176002583 after retry RPC after error@855033383) -> (continuation of Deferred@315773880 after retry RPC after error@621038020) -> (continuation of Deferred@1127198301 after retry RPC after error@1206685665) -> (continuation of Deferred@1201883959 after retry RPC after error@1089368174) -> (continuation of Deferred@2114810728 after retry RPC after error@1564305615) -> (continuation of Deferred@1735340 after retry RPC after error@868489153) -> (continuation of Deferred@642229474 after retry RPC after error@269147125) -> (continuation of Deferred@1125522498 after retry RPC after error@1358918142) -> (continuation of Deferred@1258602423 after retry RPC after error@219615313) -> (continuation of Deferred@2115041725 after retry RPC after error@1430980352) -> (continuation of Deferred@1004335897 after retry RPC after error@243071403) -> (continuation of Deferred@2002269384 after retry RPC after error@2049334820) -> (continuation of Deferred@650001986 after retry RPC after error@1340484255) -> (continuation of Deferred@2033105375 after retry RPC after error@825298848) -> (continuation of Deferred@1722444778 after retry RPC after error@417732466) -> (continuation of Deferred@344634864 after retry RPC after error@763356366) -> (continuation of Deferred@406794160 after retry RPC after error@810214978) -> (continuation of Deferred@1907286158 after retry RPC after error@589636577) -> (continuation of Deferred@1107745434 after retry RPC after error@630803566) -> (continuation of Deferred@590705347 after retry RPC after error@696608623) -> (continuation of Deferred@1571007183 after retry RPC after error@850814761) -> (continuation of Deferred@983414998 after retry RPC after error@950083317) -> (continuation of Deferred@2014588668 after retry RPC after error@2040173048))) at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) ~[main/:?] at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:413) ~[main/:?] at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:197) ~[main/:?] at org.apache.kudu.client.TestSecurity.checkClientCanReconnect(TestSecurity.java:135) ~[test/:?] at org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally(TestSecurity.java:431) ~[test/:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) ~[junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) ~[junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.12.jar:4.12] at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) ~[junit-4.12.jar:4.12] at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) ~[junit-4.12.jar:4.12] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_141] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_141] Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1676) ~[main/:?] at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2121) ~[main/:?] at org.apache.kudu.client.AsyncKuduClient.access$1200(AsyncKuduClient.java:261) ~[main/:?] at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1328) ~[main/:?] at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1318) ~[main/:?] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?] at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:289) ~[main/:?] at org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49) ~[main/:?] at org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:323) ~[main/:?] at org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:312) ~[main/:?] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?] at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275) ~[main/:?] at org.apache.kudu.client.KuduRpc.callback(KuduRpc.java:322) ~[main/:?] at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:296) ~[main/:?] at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59) ~[main/:?] at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149) ~[main/:?] at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145) ~[main/:?] at org.apache.kudu.client.Connection.messageReceived(Connection.java:390) ~[main/:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.10.6.Final.jar:?] at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238) ~[main/:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[netty-3.10.6.Final.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_141] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_141] Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873 at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:287) ~[main/:?] at org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49) ~[main/:?] at org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:323) ~[main/:?] at org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:312) ~[main/:?] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?] at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275) ~[main/:?] at org.apache.kudu.client.KuduRpc.callback(KuduRpc.java:322) ~[main/:?] at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:296) ~[main/:?] at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59) ~[main/:?] at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149) ~[main/:?] at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145) ~[main/:?] at org.apache.kudu.client.Connection.messageReceived(Connection.java:390) ~[main/:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.10.6.Final.jar:?] at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238) ~[main/:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[netty-3.10.6.Final.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_141] ... 1 more Caused by: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873 at org.apache.kudu.client.Connection.channelClosed(Connection.java:266) ~[main/:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) ~[netty-3.10.6.Final.jar:?] at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238) ~[main/:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.timeout.ReadTimeoutHandler.channelClosed(ReadTimeoutHandler.java:176) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels$6.run(Channels.java:457) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.ChannelRunnableWrapper.run(ChannelRunnableWrapper.java:40) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.6.Final.jar:?] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) ~[netty-3.10.6.Final.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_141] ... 1 more 04:09:26.218 [ERROR - main] (RetryRule.java:239) org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally: giving up after 1 attempts E04:09:26.222 [INFO - main] (RetryRule.java:136) Creating RetryStatement with result reporter and retry count of 0 (no retries) .04:09:26.223 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskusvpEt/test-tmp 04:09:26.223 [INFO - main] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin 04:09:26.223 [INFO - main] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskusvpEt/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb] 04:09:26.225 [DEBUG - main] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 0 enable_kerberos: true cluster_root: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118" extra_master_flags: "--authn_token_validity_seconds=10" hms_mode: NONE mini_kdc_options { ticket_lifetime: "10s" renew_lifetime: "20s" } } 04:09:26.420 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:26.420 [DEBUG - main] (MiniKuduCluster.java:155) Request: start_cluster { } 04:09:26.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Loading random data 04:09:26.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Initializing database '/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/principal' for realm 'KRBTEST.COM', 04:09:26.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) master key name 'K/M@KRBTEST.COM' 04:09:26.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:26 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): setting up network... 04:09:26.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:26 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) 04:09:26.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked 04:09:26.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 04:09:26.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:26 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): skipping unrecognized local address family 17 04:09:26.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:26 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): skipping unrecognized local address family 17 04:09:26.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:26 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): set up 1 sockets 04:09:26.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:26 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): commencing operation 04:09:26.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: starting... 04:09:27.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:27.007475 1285 mini_kdc.cc:119] Time spent starting KDC: real 0.559s user 0.000s sys 0.022s 04:09:27.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy 04:09:27.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:09:27.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-admin@KRBTEST.COM" created. 04:09:27.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy 04:09:27.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:09:27.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-user@KRBTEST.COM" created. 04:09:27.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy 04:09:27.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:09:27.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "joe-interloper@KRBTEST.COM" created. 04:09:27.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:09:27.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/test-user.keytab. 04:09:27.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/test-user.keytab. 04:09:27.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/test-user.keytab. 04:09:27.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/test-user.keytab. 04:09:27.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:27 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:27.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:09:27.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.1.65.126@KRBTEST.COM; defaulting to no policy 04:09:27.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:27.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.1.65.126@KRBTEST.COM" created. 04:09:27.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:27.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.65.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.126.keytab. 04:09:27.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.65.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.126.keytab. 04:09:27.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.65.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.126.keytab. 04:09:27.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.65.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.126.keytab. 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.178318 1285 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/wal 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/data 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.65.126:38761 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.65.126 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:27.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/logs 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/data/info.pb 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.126.keytab 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.65.126 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:27.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.65.126:38761,127.1.65.125:38529,127.1.65.124:43899 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:27.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:27.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:27.340576 1305 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:27.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:27.340965 1305 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.1.65.126 04:09:27.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:27.341203 1305 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:27.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:27.353184 1305 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:09:27.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:27.353355 1305 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:27.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:27.353467 1305 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:27.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:27.353556 1305 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:27.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:27.353699 1305 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:27.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:27.353817 1305 flags.cc:404] Enabled experimental flag: --principal=kudu/127.1.65.126 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.374076 1305 master_main.cc:98] Master server non-default flags: 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/data 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/wal 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.65.126:38761,127.1.65.125:38529,127.1.65.124:43899 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.65.126:38761 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.126.keytab 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.65.126 04:09:27.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/data/info.pb 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.65.126 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/logs 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:27.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:27.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.389227 1305 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 120516us 04:09:27.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:27 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608567, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.65.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:27.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.391841 1305 init.cc:359] Logged in from keytab as kudu/127.1.65.126@KRBTEST.COM (short username kudu) 04:09:27.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.395246 1305 fs_manager.cc:263] Metadata directory not provided 04:09:27.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.395417 1305 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:27.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.395704 1305 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:09:27.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.395819 1305 server_base.cc:449] Attempting to create new FS layout instead 04:09:27.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.399024 1305 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/data/instance: 04:09:27.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "536fb892576d414f90629aaf613c3a45" 04:09:27.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:27 on dist-test-slave-dist-test-slave-fvsv" 04:09:27.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.399781 1305 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/wal/instance: 04:09:27.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "536fb892576d414f90629aaf613c3a45" 04:09:27.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:27 on dist-test-slave-dist-test-slave-fvsv" 04:09:27.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.409343 1305 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s 04:09:27.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.410571 1305 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:27.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.411130 1305 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:27.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.417783 1305 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.417906 1305 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/wal 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "536fb892576d414f90629aaf613c3a45" 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:27 on dist-test-slave-dist-test-slave-fvsv" 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:27.418126 1305 fs_report.cc:352] FS layout report 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/wal 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/wal 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/data/data 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:27.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:28.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.049325 1305 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.607s user 0.199s sys 0.407s 04:09:28.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.049585 1305 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.608s user 0.199s sys 0.407s 04:09:28.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.142104 1305 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:28.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.208736 1305 rpc_server.cc:205] RPC server started. Bound to: 127.1.65.126:38761 04:09:28.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.211490 1305 webserver.cc:374] Webserver started at http://127.1.65.126:35509/ using document root and password file 04:09:28.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.212574 1305 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-0/data/info.pb 04:09:28.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.216331 1367 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:09:28.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.222761 1285 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 1305 04:09:28.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.229379 1367 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } has no permanent_uuid. Determining permanent_uuid... 04:09:28.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.1.65.125@KRBTEST.COM; defaulting to no policy 04:09:28.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:28.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.1.65.125@KRBTEST.COM" created. 04:09:28.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:28 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608567, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.65.126@KRBTEST.COM for kudu/127.1.65.126@KRBTEST.COM 04:09:28.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:28.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.65.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.125.keytab. 04:09:28.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.65.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.125.keytab. 04:09:28.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.65.125 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.125.keytab. 04:09:28.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.65.125 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.125.keytab. 04:09:28.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.270480 1285 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:28.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:28.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/wal 04:09:28.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/data 04:09:28.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:28.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.65.125:38529 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.65.125 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/logs 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/data/info.pb 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.125.keytab 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.65.125 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:28.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.65.126:38761,127.1.65.125:38529,127.1.65.124:43899 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:28.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.276245 1367 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } has no permanent_uuid. Determining permanent_uuid... 04:09:28.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.277952 1367 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.125:38529: Network error: Client connection negotiation failed: client connection to 127.1.65.125:38529: connect: Connection refused (error 111) 04:09:28.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.297370 1367 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } attempt: 1 04:09:28.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.299136 1367 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.125:38529: Network error: Client connection negotiation failed: client connection to 127.1.65.125:38529: connect: Connection refused (error 111) 04:09:28.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.347452 1367 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } attempt: 2 04:09:28.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.349201 1367 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.125:38529: Network error: Client connection negotiation failed: client connection to 127.1.65.125:38529: connect: Connection refused (error 111) 04:09:28.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:28.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.431138 1372 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:28.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.431514 1372 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.1.65.125 04:09:28.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.431756 1372 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:28.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.443397 1372 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:09:28.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.443555 1372 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:28.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.443672 1372 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:28.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.443764 1372 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:28.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.443879 1372 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:28.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.444036 1372 flags.cc:404] Enabled experimental flag: --principal=kudu/127.1.65.125 04:09:28.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.460569 1367 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } attempt: 3 04:09:28.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.463011 1367 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.125:38529: Network error: Client connection negotiation failed: client connection to 127.1.65.125:38529: connect: Connection refused (error 111) 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.464018 1372 master_main.cc:98] Master server non-default flags: 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/data 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/wal 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.65.126:38761,127.1.65.125:38529,127.1.65.124:43899 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.65.125:38529 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.125.keytab 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.65.125 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/data/info.pb 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.65.125 04:09:28.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/logs 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:28.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:28.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.478565 1372 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 121016us 04:09:28.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:28 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608568, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.65.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:28.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.480850 1372 init.cc:359] Logged in from keytab as kudu/127.1.65.125@KRBTEST.COM (short username kudu) 04:09:28.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.483867 1372 fs_manager.cc:263] Metadata directory not provided 04:09:28.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.484010 1372 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:28.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.484253 1372 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:09:28.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.484374 1372 server_base.cc:449] Attempting to create new FS layout instead 04:09:28.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.487334 1372 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/data/instance: 04:09:28.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "388574f855cf4c678c6f39df2ef5f0a8" 04:09:28.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:28 on dist-test-slave-dist-test-slave-fvsv" 04:09:28.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.488035 1372 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/wal/instance: 04:09:28.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "388574f855cf4c678c6f39df2ef5f0a8" 04:09:28.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:28 on dist-test-slave-dist-test-slave-fvsv" 04:09:28.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.497313 1372 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.003s sys 0.006s 04:09:28.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.498544 1372 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:28.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.499125 1372 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:28.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.506194 1372 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s 04:09:28.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.506326 1372 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/wal 04:09:28.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "388574f855cf4c678c6f39df2ef5f0a8" 04:09:28.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:28 on dist-test-slave-dist-test-slave-fvsv" 04:09:28.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.506544 1372 fs_report.cc:352] FS layout report 04:09:28.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:28.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/wal 04:09:28.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/wal 04:09:28.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/data/data 04:09:28.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:28.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:28.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:28.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:28.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:28.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:28.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:28.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:28.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:28.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:28.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.639396 1367 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } attempt: 4 04:09:28.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.641095 1367 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.125:38529: Network error: Client connection negotiation failed: client connection to 127.1.65.125:38529: connect: Connection refused (error 111) 04:09:28.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:28.944396 1367 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } attempt: 5 04:09:28.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:28.946154 1367 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.125:38529: Network error: Client connection negotiation failed: client connection to 127.1.65.125:38529: connect: Connection refused (error 111) 04:09:29.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.139068 1372 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.609s user 0.306s sys 0.298s 04:09:29.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.139292 1372 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.609s user 0.306s sys 0.299s 04:09:29.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.175653 1372 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:29.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.238730 1372 rpc_server.cc:205] RPC server started. Bound to: 127.1.65.125:38529 04:09:29.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.241674 1372 webserver.cc:374] Webserver started at http://127.1.65.125:36845/ using document root and password file 04:09:29.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.242714 1372 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-1/data/info.pb 04:09:29.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.244863 1285 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 1372 04:09:29.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.246657 1434 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:09:29.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.1.65.124@KRBTEST.COM; defaulting to no policy 04:09:29.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:29.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.1.65.124@KRBTEST.COM" created. 04:09:29.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.266221 1434 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } has no permanent_uuid. Determining permanent_uuid... 04:09:29.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:29 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608568, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.65.125@KRBTEST.COM for kudu/127.1.65.126@KRBTEST.COM 04:09:29.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.302170 1434 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } has no permanent_uuid. Determining permanent_uuid... 04:09:29.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:29.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.65.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.124.keytab. 04:09:29.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.65.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.124.keytab. 04:09:29.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.65.124 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.124.keytab. 04:09:29.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.65.124 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.124.keytab. 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.306985 1285 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/wal 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/data 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.65.124:43899 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.65.124 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/logs 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/data/info.pb 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:29.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:29.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:29.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:29.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.124.keytab 04:09:29.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.65.124 04:09:29.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:29.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:29.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:29.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:29.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:29.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.65.126:38761,127.1.65.125:38529,127.1.65.124:43899 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:29.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:29 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608568, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.65.125@KRBTEST.COM for kudu/127.1.65.125@KRBTEST.COM 04:09:29.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.322661 1434 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } has no permanent_uuid. Determining permanent_uuid... 04:09:29.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.324190 1434 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.124:43899: Network error: Client connection negotiation failed: client connection to 127.1.65.124:43899: connect: Connection refused (error 111) 04:09:29.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.343552 1434 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } attempt: 1 04:09:29.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.345232 1434 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.124:43899: Network error: Client connection negotiation failed: client connection to 127.1.65.124:43899: connect: Connection refused (error 111) 04:09:29.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.393573 1434 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } attempt: 2 04:09:29.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.395464 1434 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.124:43899: Network error: Client connection negotiation failed: client connection to 127.1.65.124:43899: connect: Connection refused (error 111) 04:09:29.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:29.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.451675 1440 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:29.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.452009 1440 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.1.65.124 04:09:29.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.452231 1440 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:29.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.463968 1440 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:09:29.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.464113 1440 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:29.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.464217 1440 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:29.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.464304 1440 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:29.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.464417 1440 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:29.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.464541 1440 flags.cc:404] Enabled experimental flag: --principal=kudu/127.1.65.124 04:09:29.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.480459 1367 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } attempt: 6 04:09:29.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.483649 1440 master_main.cc:98] Master server non-default flags: 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/data 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/wal 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.65.126:38761,127.1.65.125:38529,127.1.65.124:43899 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.65.124:43899 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kudu_127.1.65.124.keytab 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.65.124 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/data/info.pb 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.65.124 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:29.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:29.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:29.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:29.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/logs 04:09:29.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:29.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:29.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:29.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:29.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:29.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:29.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:29.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:29.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:29.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:29 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608567, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.65.126@KRBTEST.COM for kudu/127.1.65.125@KRBTEST.COM 04:09:29.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.499121 1367 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } has no permanent_uuid. Determining permanent_uuid... 04:09:29.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.500694 1367 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.124:43899: Network error: Client connection negotiation failed: client connection to 127.1.65.124:43899: connect: Connection refused (error 111) 04:09:29.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.501668 1440 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 121516us 04:09:29.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:29 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.65.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:29.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.503903 1440 init.cc:359] Logged in from keytab as kudu/127.1.65.124@KRBTEST.COM (short username kudu) 04:09:29.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.506809 1440 fs_manager.cc:263] Metadata directory not provided 04:09:29.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.506784 1434 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } attempt: 3 04:09:29.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.507006 1440 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:29.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.507397 1440 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:09:29.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.507524 1440 server_base.cc:449] Attempting to create new FS layout instead 04:09:29.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.508321 1434 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.124:43899: Network error: Client connection negotiation failed: client connection to 127.1.65.124:43899: connect: Connection refused (error 111) 04:09:29.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.510711 1440 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/data/instance: 04:09:29.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" 04:09:29.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:29 on dist-test-slave-dist-test-slave-fvsv" 04:09:29.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.511507 1440 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/wal/instance: 04:09:29.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" 04:09:29.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:29 on dist-test-slave-dist-test-slave-fvsv" 04:09:29.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.520694 1440 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s 04:09:29.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.521912 1440 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:29.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.522418 1440 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.529130 1440 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.529256 1440 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/wal 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:29 on dist-test-slave-dist-test-slave-fvsv" 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.529474 1440 fs_report.cc:352] FS layout report 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/wal 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/wal 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/data/data 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:29.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:29.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:29.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:29.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:29.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:29.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:29.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:29.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.532999 1367 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } attempt: 1 04:09:29.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.534659 1367 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.124:43899: Network error: Client connection negotiation failed: client connection to 127.1.65.124:43899: connect: Connection refused (error 111) 04:09:29.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.603142 1367 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } attempt: 2 04:09:29.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.605320 1367 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.124:43899: Network error: Client connection negotiation failed: client connection to 127.1.65.124:43899: connect: Connection refused (error 111) 04:09:29.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.681620 1367 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } attempt: 3 04:09:29.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.683684 1367 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.124:43899: Network error: Client connection negotiation failed: client connection to 127.1.65.124:43899: connect: Connection refused (error 111) 04:09:29.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.684679 1434 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } attempt: 4 04:09:29.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.686650 1434 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.124:43899: Network error: Client connection negotiation failed: client connection to 127.1.65.124:43899: connect: Connection refused (error 111) 04:09:29.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.821002 1367 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } attempt: 4 04:09:29.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.823120 1367 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.124:43899: Network error: Client connection negotiation failed: client connection to 127.1.65.124:43899: connect: Connection refused (error 111) 04:09:29.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:29.989977 1434 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } attempt: 5 04:09:29.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:29.992125 1434 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.124:43899: Network error: Client connection negotiation failed: client connection to 127.1.65.124:43899: connect: Connection refused (error 111) 04:09:30.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.106437 1367 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } attempt: 5 04:09:30.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:30.108283 1367 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.65.124:43899: Network error: Client connection negotiation failed: client connection to 127.1.65.124:43899: connect: Connection refused (error 111) 04:09:30.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:30.175225 1440 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.622s user 0.242s sys 0.376s 04:09:30.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:30.175421 1440 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.622s user 0.242s sys 0.376s 04:09:30.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.214735 1440 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:30.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.283934 1440 rpc_server.cc:205] RPC server started. Bound to: 127.1.65.124:43899 04:09:30.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.286607 1440 webserver.cc:374] Webserver started at http://127.1.65.124:37905/ using document root and password file 04:09:30.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.287612 1440 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/master-2/data/info.pb 04:09:30.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.290608 1285 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 1440 04:09:30.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.291990 1503 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:09:30.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.305523 1503 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } has no permanent_uuid. Determining permanent_uuid... 04:09:30.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.65.124@KRBTEST.COM for kudu/127.1.65.126@KRBTEST.COM 04:09:30.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.65.124@KRBTEST.COM 04:09:30.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.333545 1503 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } has no permanent_uuid. Determining permanent_uuid... 04:09:30.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.65.124@KRBTEST.COM for kudu/127.1.65.125@KRBTEST.COM 04:09:30.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.65.125@KRBTEST.COM 04:09:30.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.356252 1503 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } has no permanent_uuid. Determining permanent_uuid... 04:09:30.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608569, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.65.124@KRBTEST.COM for kudu/127.1.65.124@KRBTEST.COM 04:09:30.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.65.126@KRBTEST.COM 04:09:30.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.374620 1285 external_mini_cluster.cc:628] 0 TS(s) registered with all masters 04:09:30.374 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:30.375 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_kdc_env_vars { } 04:09:30.376 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_kdc_env_vars { env_vars { key: "KRB5CCNAME" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/krb5cc" } env_vars { key: "KUDU_ENABLE_KRB5_REALM_FIX" value: "yes" } env_vars { key: "KRB5_KDC_PROFILE" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/kdc.conf" } env_vars { key: "KRB5_CONFIG" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/krb5.conf" } } 04:09:30.377 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_masters { } 04:09:30.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.379500 1503 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce: Bootstrap starting. 04:09:30.377 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.1.65.126" port: 38761 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.1.65.125" port: 38529 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.1.65.124" port: 43899 } } } 04:09:30.380 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_tservers { } 04:09:30.380 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:30.380 [INFO - main] (MiniKuduCluster.java:439) Running kinit for user test-admin 04:09:30.380 [DEBUG - main] (MiniKuduCluster.java:155) Request: kinit { username: "test-admin" } 04:09:30.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.382635 1503 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce: No blocks or log segments found. Creating new log. 04:09:30.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.384218 1503 log.cc:525] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce: Log is configured to *not* fsync() on all Append() calls 04:09:30.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.392689 1503 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce: No bootstrap required, opened a new log 04:09:30.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:30.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:09:30.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.413632 1503 raft_consensus.cc:340] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } 04:09:30.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.414371 1503 raft_consensus.cc:366] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:09:30.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.414582 1503 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6acc0b69a8cf4a0cb3afea62564b39ce, State: Initialized, Role: FOLLOWER 04:09:30.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.415258 1503 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } 04:09:30.417 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:30.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.418169 1503 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.418820 1503 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [sys.catalog]: This master's current role is: FOLLOWER 04:09:30.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.419888 1503 sys_catalog.cc:427] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [sys.catalog]: configured and running, proceeding with master startup. 04:09:30.431 [DEBUG - main] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/krb5cc 04:09:30.433 [DEBUG - main] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:09:30.441 [DEBUG - New I/O worker #21] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:09:30.442 [DEBUG - New I/O worker #21] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/krb5cc 04:09:30.444 [DEBUG - New I/O worker #21] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:09:30.444 [INFO - New I/O worker #21] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:09:30.445 [DEBUG - New I/O worker #20] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:09:30.445 [DEBUG - New I/O worker #20] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/krb5cc 04:09:30.450 [DEBUG - New I/O worker #21] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.1.65.124 04:09:30.451 [DEBUG - New I/O worker #20] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:09:30.452 [INFO - New I/O worker #20] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:09:30.452 [DEBUG - New I/O worker #20] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.1.65.126 04:09:30.453 [DEBUG - New I/O worker #19] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:09:30.453 [DEBUG - New I/O worker #19] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6112081972096341118/krb5kdc/krb5cc 04:09:30.454 [DEBUG - New I/O worker #19] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:09:30.455 [INFO - New I/O worker #19] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:09:30.455 [DEBUG - New I/O worker #19] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.1.65.125 04:09:30.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.65.125@KRBTEST.COM 04:09:30.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.65.124@KRBTEST.COM 04:09:30.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.65.126@KRBTEST.COM 04:09:30.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.418401 1513 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.500118 1513 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [sys.catalog]: This master's current role is: FOLLOWER 04:09:30.501 [DEBUG - New I/O worker #19] (Negotiator.java:764) Authenticated connection [id: 0xa929d602, /127.0.0.1:51582 => /127.1.65.125:38529] using SASL/GSSAPI 04:09:30.506 [INFO - New I/O worker #19] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.125:38529: Catalog manager is not initialized. State: Starting 04:09:30.508 [DEBUG - New I/O worker #20] (Negotiator.java:764) Authenticated connection [id: 0x44e09685, /127.0.0.1:45580 => /127.1.65.126:38761] using SASL/GSSAPI 04:09:30.509 [DEBUG - New I/O worker #21] (Negotiator.java:764) Authenticated connection [id: 0xf21a9ff6, /127.0.0.1:43194 => /127.1.65.124:43899] using SASL/GSSAPI 04:09:30.511 [INFO - New I/O worker #20] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.126:38761: Catalog manager is not initialized. State: Starting 04:09:30.512 [INFO - New I/O worker #21] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.124:43899: Catalog manager is not initialized. State: Starting 04:09:30.513 [WARN - New I/O worker #21] (ConnectToCluster.java:281) Unable to find the leader master 127.1.65.125:38529,127.1.65.126:38761,127.1.65.124:43899; will retry 04:09:30.518 [INFO - New I/O worker #21] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.124:43899: Catalog manager is not initialized. State: Starting 04:09:30.519 [INFO - New I/O worker #19] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.125:38529: Catalog manager is not initialized. State: Starting 04:09:30.520 [INFO - New I/O worker #20] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.126:38761: Catalog manager is not initialized. State: Starting 04:09:30.520 [WARN - New I/O worker #20] (ConnectToCluster.java:281) Unable to find the leader master 127.1.65.125:38529,127.1.65.126:38761,127.1.65.124:43899; will retry 04:09:30.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.526417 1434 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } attempt: 6 04:09:30.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608568, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.65.125@KRBTEST.COM for kudu/127.1.65.124@KRBTEST.COM 04:09:30.537 [INFO - New I/O worker #21] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.124:43899: Catalog manager is not initialized. State: Starting 04:09:30.539 [INFO - New I/O worker #20] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.126:38761: Catalog manager is not initialized. State: Starting 04:09:30.539 [INFO - New I/O worker #19] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.125:38529: Catalog manager is not initialized. State: Starting 04:09:30.540 [WARN - New I/O worker #19] (ConnectToCluster.java:281) Unable to find the leader master 127.1.65.125:38529,127.1.65.126:38761,127.1.65.124:43899; will retry 04:09:30.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:30.469759 1459 tablet.cc:1799] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:09:30.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:30.552320 1516 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:30.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:30.552506 1516 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:30.558 [INFO - New I/O worker #19] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.125:38529: Catalog manager is not initialized. State: Starting 04:09:30.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.557147 1434 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8: Bootstrap starting. 04:09:30.561 [INFO - New I/O worker #20] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.126:38761: Catalog manager is not initialized. State: Starting 04:09:30.561 [WARN - New I/O worker #20] (ConnectToCluster.java:281) Unable to find the leader master 127.1.65.125:38529,127.1.65.126:38761,127.1.65.124:43899; will retry 04:09:30.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.562285 1434 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8: No blocks or log segments found. Creating new log. 04:09:30.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.564998 1434 log.cc:525] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8: Log is configured to *not* fsync() on all Append() calls 04:09:30.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.568682 1434 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8: No bootstrap required, opened a new log 04:09:30.578 [INFO - New I/O worker #19] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.125:38529: Catalog manager is not initialized. State: Starting 04:09:30.580 [INFO - New I/O worker #20] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.126:38761: Catalog manager is not initialized. State: Starting 04:09:30.580 [WARN - New I/O worker #20] (ConnectToCluster.java:281) Unable to find the leader master 127.1.65.125:38529,127.1.65.126:38761,127.1.65.124:43899; will retry 04:09:30.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.583835 1434 raft_consensus.cc:340] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } 04:09:30.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.584355 1434 raft_consensus.cc:366] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:09:30.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.584492 1434 raft_consensus.cc:705] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 388574f855cf4c678c6f39df2ef5f0a8, State: Initialized, Role: FOLLOWER 04:09:30.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.584969 1434 consensus_queue.cc:229] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } 04:09:30.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.587793 1528 sys_catalog.cc:339] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.588270 1528 sys_catalog.cc:342] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [sys.catalog]: This master's current role is: FOLLOWER 04:09:30.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.588577 1434 sys_catalog.cc:339] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.588960 1434 sys_catalog.cc:342] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [sys.catalog]: This master's current role is: FOLLOWER 04:09:30.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.589818 1434 sys_catalog.cc:427] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [sys.catalog]: configured and running, proceeding with master startup. 04:09:30.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.592211 1529 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:09:30.620 [INFO - New I/O worker #20] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.126:38761: Catalog manager is not initialized. State: Starting 04:09:30.620 [WARN - New I/O worker #20] (ConnectToCluster.java:281) Unable to find the leader master 127.1.65.125:38529,127.1.65.126:38761,127.1.65.124:43899; will retry 04:09:30.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.620587 1367 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } attempt: 6 04:09:30.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[1294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608567, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.65.126@KRBTEST.COM for kudu/127.1.65.124@KRBTEST.COM 04:09:30.638 [INFO - New I/O worker #20] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.126:38761: Catalog manager is not initialized. State: Starting 04:09:30.639 [WARN - New I/O worker #21] (ConnectToCluster.java:281) Unable to find the leader master 127.1.65.125:38529,127.1.65.126:38761,127.1.65.124:43899; will retry 04:09:30.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.650230 1367 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45: Bootstrap starting. 04:09:30.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.655382 1367 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45: No blocks or log segments found. Creating new log. 04:09:30.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.656775 1367 log.cc:525] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45: Log is configured to *not* fsync() on all Append() calls 04:09:30.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.664934 1367 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45: No bootstrap required, opened a new log 04:09:30.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.681161 1367 raft_consensus.cc:340] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } 04:09:30.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.681708 1367 raft_consensus.cc:366] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:09:30.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.681843 1367 raft_consensus.cc:705] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 536fb892576d414f90629aaf613c3a45, State: Initialized, Role: FOLLOWER 04:09:30.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.682345 1367 consensus_queue.cc:229] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } 04:09:30.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.688606 1367 sys_catalog.cc:339] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.689287 1367 sys_catalog.cc:342] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [sys.catalog]: This master's current role is: FOLLOWER 04:09:30.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.690243 1367 sys_catalog.cc:427] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [sys.catalog]: configured and running, proceeding with master startup. 04:09:30.701 [INFO - New I/O worker #20] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.126:38761: Catalog manager is not initialized. State: Starting 04:09:30.705 [WARN - New I/O worker #21] (ConnectToCluster.java:281) Unable to find the leader master 127.1.65.125:38529,127.1.65.126:38761,127.1.65.124:43899; will retry 04:09:30.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.750116 1513 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:09:30.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.750455 1513 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } 04:09:30.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.754755 1513 leader_election.cc:296] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 536fb892576d414f90629aaf613c3a45 (127.1.65.126:38761), 388574f855cf4c678c6f39df2ef5f0a8 (127.1.65.125:38529) 04:09:30.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.756466 1411 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "388574f855cf4c678c6f39df2ef5f0a8" is_pre_election: true 04:09:30.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.757076 1411 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6acc0b69a8cf4a0cb3afea62564b39ce in term 0. 04:09:30.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.760525 1451 leader_election.cc:310] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 388574f855cf4c678c6f39df2ef5f0a8, 6acc0b69a8cf4a0cb3afea62564b39ce; no voters: 04:09:30.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.761327 1513 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [term 0 FOLLOWER]: Leader pre-election won for term 1 04:09:30.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.761485 1513 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:09:30.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.761618 1513 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [term 0 FOLLOWER]: Advancing to term 1 04:09:30.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.766271 1513 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } 04:09:30.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.770686 1411 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "388574f855cf4c678c6f39df2ef5f0a8" 04:09:30.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.771229 1411 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [term 0 FOLLOWER]: Advancing to term 1 04:09:30.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:30.681715 1390 tablet.cc:1799] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:09:30.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.776190 1513 leader_election.cc:296] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [CANDIDATE]: Term 1 election: Requested vote from peers 536fb892576d414f90629aaf613c3a45 (127.1.65.126:38761), 388574f855cf4c678c6f39df2ef5f0a8 (127.1.65.125:38529) 04:09:30.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.777556 1411 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6acc0b69a8cf4a0cb3afea62564b39ce in term 1. 04:09:30.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.778479 1451 leader_election.cc:310] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 388574f855cf4c678c6f39df2ef5f0a8, 6acc0b69a8cf4a0cb3afea62564b39ce; no voters: 04:09:30.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.779078 1513 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [term 1 FOLLOWER]: Leader election won for term 1 04:09:30.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.782593 1513 raft_consensus.cc:667] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [term 1 LEADER]: Becoming Leader. State: Replica: 6acc0b69a8cf4a0cb3afea62564b39ce, State: Running, Role: LEADER 04:09:30.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.783241 1513 consensus_queue.cc:206] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } 04:09:30.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.688850 1532 sys_catalog.cc:339] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.786911 1532 sys_catalog.cc:342] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [sys.catalog]: This master's current role is: FOLLOWER 04:09:30.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.792325 1534 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6acc0b69a8cf4a0cb3afea62564b39ce. Latest consensus state: current_term: 1 leader_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.794210 1534 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [sys.catalog]: This master's current role is: LEADER 04:09:30.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.798483 1536 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:09:30.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.800865 1536 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:09:30.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.754433 1344 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536fb892576d414f90629aaf613c3a45" is_pre_election: true 04:09:30.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.822738 1344 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6acc0b69a8cf4a0cb3afea62564b39ce in term 0. 04:09:30.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.769724 1343 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536fb892576d414f90629aaf613c3a45" 04:09:30.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.825861 1343 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [term 0 FOLLOWER]: Advancing to term 1 04:09:30.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:30.830766 1533 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:30.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:30.831156 1533 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:30.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.831351 1343 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6acc0b69a8cf4a0cb3afea62564b39ce in term 1. 04:09:30.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.869016 1411 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [term 1 FOLLOWER]: Refusing update from remote peer 6acc0b69a8cf4a0cb3afea62564b39ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:09:30.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.870760 1343 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [term 1 FOLLOWER]: Refusing update from remote peer 6acc0b69a8cf4a0cb3afea62564b39ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:09:30.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.872014 1534 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:09:30.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.872943 1513 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:09:30.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.892977 1528 sys_catalog.cc:339] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6acc0b69a8cf4a0cb3afea62564b39ce. Latest consensus state: current_term: 1 leader_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.893426 1528 sys_catalog.cc:342] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [sys.catalog]: This master's current role is: FOLLOWER 04:09:30.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.897184 1538 mvcc.cc:203] Tried to move safe_time back from 6396348706250575872 to 6396348705933815808. Current Snapshot: MvccSnapshot[committed={T|T < 6396348706250575872}] 04:09:30.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:30.900660 1323 tablet.cc:1799] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:09:30.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.901144 1539 mvcc.cc:203] Tried to move safe_time back from 6396348706250575872 to 6396348705933815808. Current Snapshot: MvccSnapshot[committed={T|T < 6396348706250575872}] 04:09:30.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.901566 1532 sys_catalog.cc:339] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6acc0b69a8cf4a0cb3afea62564b39ce. Latest consensus state: current_term: 1 leader_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.902074 1532 sys_catalog.cc:342] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [sys.catalog]: This master's current role is: FOLLOWER 04:09:30.903 [WARN - New I/O worker #20] (ConnectToCluster.java:278) None of the provided masters 127.1.65.125:38529,127.1.65.126:38761,127.1.65.124:43899 is a leader; will retry 04:09:30.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.903909 1513 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.904484 1513 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [sys.catalog]: This master's current role is: LEADER 04:09:30.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.906961 1528 sys_catalog.cc:339] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.907469 1528 sys_catalog.cc:342] T 00000000000000000000000000000000 P 388574f855cf4c678c6f39df2ef5f0a8 [sys.catalog]: This master's current role is: FOLLOWER 04:09:30.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.909114 1534 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.909727 1534 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce [sys.catalog]: This master's current role is: LEADER 04:09:30.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.909791 1536 catalog_manager.cc:959] Generated new certificate authority record 04:09:30.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.911432 1536 catalog_manager.cc:1084] Loading token signing keys... 04:09:30.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.913166 1532 sys_catalog.cc:339] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536fb892576d414f90629aaf613c3a45" member_type: VOTER last_known_addr { host: "127.1.65.126" port: 38761 } } peers { permanent_uuid: "388574f855cf4c678c6f39df2ef5f0a8" member_type: VOTER last_known_addr { host: "127.1.65.125" port: 38529 } } peers { permanent_uuid: "6acc0b69a8cf4a0cb3afea62564b39ce" member_type: VOTER last_known_addr { host: "127.1.65.124" port: 43899 } } } 04:09:30.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.913600 1532 sys_catalog.cc:342] T 00000000000000000000000000000000 P 536fb892576d414f90629aaf613c3a45 [sys.catalog]: This master's current role is: FOLLOWER 04:09:30.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:30.947691 1536 catalog_manager.cc:4350] T 00000000000000000000000000000000 P 6acc0b69a8cf4a0cb3afea62564b39ce: Generated new TSK 0 04:09:31.277 [DEBUG - New I/O worker #21] (AsyncKuduClient.java:2322) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 04:09:31.345 [DEBUG - main] (SecurityUtil.java:121) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 04:09:31.346 [DEBUG - main] (SecurityContext.java:312) Importing authentication credentials with one authn token, 0 cert(s), and realUser=slave 04:09:31.358 [DEBUG - New I/O worker #29] (Connection.java:691) [peer master-127.1.65.126:38761(127.1.65.126:38761)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client 04:09:31.358 [INFO - New I/O worker #29] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.126:38761: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client 04:09:31.360 [DEBUG - New I/O worker #28] (Connection.java:691) [peer master-127.1.65.125:38529(127.1.65.125:38529)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client 04:09:31.360 [INFO - New I/O worker #28] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.125:38529: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client 04:09:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:31.364370 1557 negotiation.cc:313] Failed RPC negotiation. Trace: 04:09:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.348322 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:51592 04:09:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357267 (+ 8945us) server_negotiation.cc:184] Beginning negotiation 04:09:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357283 (+ 16us) server_negotiation.cc:373] Waiting for connection header 04:09:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357315 (+ 32us) server_negotiation.cc:381] Connection header received 04:09:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357417 (+ 102us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:09:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357430 (+ 13us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:09:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357523 (+ 93us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:09:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.363314 (+ 5791us) server_negotiation.cc:205] Negotiated authn=SASL 04:09:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.363683 (+ 369us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51592: BlockingRecv error: recv got EOF from 127.0.0.1:51592 (error 108) 04:09:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":8724,"thread_start_us":2963,"threads_started":1} 04:09:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:31.367244 1558 negotiation.cc:313] Failed RPC negotiation. Trace: 04:09:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.350366 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:45590 04:09:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357244 (+ 6878us) server_negotiation.cc:184] Beginning negotiation 04:09:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357265 (+ 21us) server_negotiation.cc:373] Waiting for connection header 04:09:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357303 (+ 38us) server_negotiation.cc:381] Connection header received 04:09:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357446 (+ 143us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:09:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357468 (+ 22us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:09:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357587 (+ 119us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:09:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.357769 (+ 182us) server_negotiation.cc:205] Negotiated authn=SASL 04:09:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.362501 (+ 4732us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45590: BlockingRecv error: recv got EOF from 127.0.0.1:45590 (error 108) 04:09:31.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":6567,"thread_start_us":8624,"threads_started":1} 04:09:31.368 [DEBUG - New I/O worker #30] (Connection.java:691) [peer master-127.1.65.124:43899(127.1.65.124:43899)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client 04:09:31.369 [INFO - New I/O worker #30] (ConnectToCluster.java:361) Unable to connect to master 127.1.65.124:43899: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client 04:09:31.369 [DEBUG - New I/O worker #30] (AsyncKuduClient.java:1331) Notify RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=22), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.65.125:38529, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.65.126:38761, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.65.124:43899, [11ms] Sub RPC ConnectToMaster: received response from server master-127.1.65.126:38761: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client, [13ms] Sub RPC ConnectToMaster: received response from server master-127.1.65.125:38529: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client, [22ms] Sub RPC ConnectToMaster: received response from server master-127.1.65.124:43899: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client, deferred=null) after lookup exception org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.1.65.125:38529,127.1.65.126:38761,127.1.65.124:43899). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client] at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:244) [main/:?] at org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49) [main/:?] at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:363) [main/:?] at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:352) [main/:?] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?] at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275) [main/:?] at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:329) [main/:?] at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:247) [main/:?] at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59) [main/:?] at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149) [main/:?] at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145) [main/:?] at org.apache.kudu.client.Connection.cleanup(Connection.java:719) [main/:?] at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:464) [main/:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [netty-3.10.6.Final.jar:?] at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238) [main/:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:?] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:?] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141] 04:09:31.370 [DEBUG - New I/O worker #30] (KuduRpc.java:258) Handling a callback on RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=23), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.65.125:38529, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.65.126:38761, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.65.124:43899, [11ms] Sub RPC ConnectToMaster: received response from server master-127.1.65.126:38761: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client, [13ms] Sub RPC ConnectToMaster: received response from server master-127.1.65.125:38529: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client, [22ms] Sub RPC ConnectToMaster: received response from server master-127.1.65.124:43899: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no TLS certificates are trusted by the client, deferred=null) with no deferred attached! 04:09:31.376 [DEBUG - main] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:09:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:31.378162 1559 negotiation.cc:313] Failed RPC negotiation. Trace: 04:09:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.351856 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:43204 04:09:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.356186 (+ 4330us) server_negotiation.cc:184] Beginning negotiation 04:09:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.356214 (+ 28us) server_negotiation.cc:373] Waiting for connection header 04:09:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.356270 (+ 56us) server_negotiation.cc:381] Connection header received 04:09:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.368335 (+ 12065us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:09:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.368355 (+ 20us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:09:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.368453 (+ 98us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:09:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.376925 (+ 8472us) server_negotiation.cc:205] Negotiated authn=SASL 04:09:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:31.377415 (+ 490us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43204: BlockingRecv error: recv got EOF from 127.0.0.1:43204 (error 108) 04:09:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":3945,"thread_start_us":3655,"threads_started":1} 04:09:31.390 [DEBUG - New I/O worker #20] (Connection.java:691) [peer master-127.1.65.126:38761(127.1.65.126:38761)] cleaning up while in state READY due to: connection disconnected 04:09:31.395 [DEBUG - New I/O worker #21] (Connection.java:691) [peer master-127.1.65.124:43899(127.1.65.124:43899)] cleaning up while in state READY due to: connection disconnected 04:09:31.397 [DEBUG - New I/O worker #19] (Connection.java:691) [peer master-127.1.65.125:38529(127.1.65.125:38529)] cleaning up while in state READY due to: connection disconnected 04:09:31.398 [DEBUG - main] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:09:31.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:31.407477 1285 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 1305 04:09:31.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:31.432633 1285 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 1372 04:09:31.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:31.449827 1285 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 1440 04:09:32.491 [INFO - main] (RetryRule.java:136) Creating RetryStatement with result reporter and retry count of 0 (no retries) .04:09:32.492 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskusvpEt/test-tmp 04:09:32.492 [INFO - main] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin 04:09:32.492 [INFO - main] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskusvpEt/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb] 04:09:32.494 [DEBUG - main] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 0 enable_kerberos: true cluster_root: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654" extra_master_flags: "--leader_failure_max_missed_heartbeat_periods=10.0" hms_mode: NONE mini_kdc_options { } } 04:09:32.674 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:32.674 [DEBUG - main] (MiniKuduCluster.java:155) Request: start_cluster { } 04:09:32.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Loading random data 04:09:32.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Initializing database '/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/principal' for realm 'KRBTEST.COM', 04:09:32.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) master key name 'K/M@KRBTEST.COM' 04:09:32.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:32 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): setting up network... 04:09:32.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:32 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) 04:09:32.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked 04:09:32.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 04:09:32.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:32 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): skipping unrecognized local address family 17 04:09:32.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:32 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): skipping unrecognized local address family 17 04:09:32.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:32 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): set up 1 sockets 04:09:32.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:32 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): commencing operation 04:09:32.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: starting... 04:09:33.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:33.251139 1563 mini_kdc.cc:119] Time spent starting KDC: real 0.550s user 0.001s sys 0.015s 04:09:33.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy 04:09:33.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:09:33.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-admin@KRBTEST.COM" created. 04:09:33.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy 04:09:33.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:09:33.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-user@KRBTEST.COM" created. 04:09:33.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy 04:09:33.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:09:33.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "joe-interloper@KRBTEST.COM" created. 04:09:33.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:09:33.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/test-user.keytab. 04:09:33.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/test-user.keytab. 04:09:33.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/test-user.keytab. 04:09:33.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/test-user.keytab. 04:09:33.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:33 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:33.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:09:33.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.1.134.254@KRBTEST.COM; defaulting to no policy 04:09:33.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:33.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.1.134.254@KRBTEST.COM" created. 04:09:33.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:33.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.254.keytab. 04:09:33.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.254.keytab. 04:09:33.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.134.254 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.254.keytab. 04:09:33.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.134.254 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.254.keytab. 04:09:33.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.415555 1563 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:33.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:33.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/wal 04:09:33.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data 04:09:33.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:33.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:33.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.134.254:46663 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.134.254 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/logs 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data/info.pb 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.254.keytab 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.134.254 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --leader_failure_max_missed_heartbeat_periods=10.0 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:33.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.134.254:46663,127.1.134.253:34103,127.1.134.252:45751 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:33.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:33.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:33.561738 1583 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:33.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:33.562081 1583 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.1.134.254 04:09:33.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:33.562312 1583 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:33.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:33.573107 1583 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:33.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:33.573266 1583 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:33.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:33.573360 1583 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:33.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:33.573470 1583 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:33.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:33.573595 1583 flags.cc:404] Enabled experimental flag: --principal=kudu/127.1.134.254 04:09:33.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.591565 1583 master_main.cc:98] Master server non-default flags: 04:09:33.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --leader_failure_max_missed_heartbeat_periods=10 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/wal 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.134.254:46663,127.1.134.253:34103,127.1.134.252:45751 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.134.254:46663 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.254.keytab 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.134.254 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data/info.pb 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.134.254 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/logs 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:33.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:33.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:33.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:33.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:33.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:33.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:33.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.609025 1583 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 123516us 04:09:33.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:33 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:33.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.611591 1583 init.cc:359] Logged in from keytab as kudu/127.1.134.254@KRBTEST.COM (short username kudu) 04:09:33.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.614656 1583 fs_manager.cc:263] Metadata directory not provided 04:09:33.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.614776 1583 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:33.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.615036 1583 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:09:33.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.615202 1583 server_base.cc:449] Attempting to create new FS layout instead 04:09:33.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.618064 1583 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data/instance: 04:09:33.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "35094e87862c4278baa1b66918879cc2" 04:09:33.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:33 on dist-test-slave-dist-test-slave-fvsv" 04:09:33.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.618777 1583 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/wal/instance: 04:09:33.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "35094e87862c4278baa1b66918879cc2" 04:09:33.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:33 on dist-test-slave-dist-test-slave-fvsv" 04:09:33.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.627754 1583 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s 04:09:33.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.629269 1583 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:33.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.629767 1583 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:33.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.636684 1583 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.636812 1583 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/wal 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "35094e87862c4278baa1b66918879cc2" 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:33 on dist-test-slave-dist-test-slave-fvsv" 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:33.637034 1583 fs_report.cc:352] FS layout report 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/wal 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/wal 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data/data 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:33.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:33.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:34.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.269567 1583 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.598s user 0.001s sys 0.001s 04:09:34.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.269831 1583 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.599s user 0.001s sys 0.001s 04:09:34.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.294898 1583 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:34.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.366722 1583 rpc_server.cc:205] RPC server started. Bound to: 127.1.134.254:46663 04:09:34.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.369622 1583 webserver.cc:374] Webserver started at http://127.1.134.254:40647/ using document root and password file 04:09:34.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.370512 1583 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data/info.pb 04:09:34.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.374745 1645 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:09:34.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.378448 1563 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 1583 04:09:34.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.392642 1645 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } has no permanent_uuid. Determining permanent_uuid... 04:09:34.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.1.134.253@KRBTEST.COM; defaulting to no policy 04:09:34.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:34.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.1.134.253@KRBTEST.COM" created. 04:09:34.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:34 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.254@KRBTEST.COM for kudu/127.1.134.254@KRBTEST.COM 04:09:34.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.427326 1645 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } has no permanent_uuid. Determining permanent_uuid... 04:09:34.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.428943 1645 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.253:34103: Network error: Client connection negotiation failed: client connection to 127.1.134.253:34103: connect: Connection refused (error 111) 04:09:34.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:34.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.134.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.253.keytab. 04:09:34.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.134.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.253.keytab. 04:09:34.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.134.253 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.253.keytab. 04:09:34.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.134.253 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.253.keytab. 04:09:34.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.430320 1563 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:34.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:34.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/wal 04:09:34.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data 04:09:34.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:34.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:34.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:34.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.134.253:34103 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.134.253 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/logs 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data/info.pb 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.253.keytab 04:09:34.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.134.253 04:09:34.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:34.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:34.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:34.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --leader_failure_max_missed_heartbeat_periods=10.0 04:09:34.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:34.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.134.254:46663,127.1.134.253:34103,127.1.134.252:45751 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:34.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.448282 1645 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } attempt: 1 04:09:34.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.449970 1645 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.253:34103: Network error: Client connection negotiation failed: client connection to 127.1.134.253:34103: connect: Connection refused (error 111) 04:09:34.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.498273 1645 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } attempt: 2 04:09:34.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.500130 1645 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.253:34103: Network error: Client connection negotiation failed: client connection to 127.1.134.253:34103: connect: Connection refused (error 111) 04:09:34.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:34.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.576932 1650 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:34.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.577270 1650 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.1.134.253 04:09:34.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.577494 1650 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:34.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.588572 1650 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:34.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.588726 1650 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:34.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.588819 1650 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:34.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.588922 1650 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:34.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.589035 1650 flags.cc:404] Enabled experimental flag: --principal=kudu/127.1.134.253 04:09:34.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.608201 1650 master_main.cc:98] Master server non-default flags: 04:09:34.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --leader_failure_max_missed_heartbeat_periods=10 04:09:34.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data 04:09:34.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/wal 04:09:34.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:34.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.134.254:46663,127.1.134.253:34103,127.1.134.252:45751 04:09:34.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:34.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:34.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.134.253:34103 04:09:34.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:34.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:34.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.253.keytab 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.134.253 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data/info.pb 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.134.253 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/logs 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:34.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:34.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.611413 1645 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } attempt: 3 04:09:34.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.613003 1645 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.253:34103: Network error: Client connection negotiation failed: client connection to 127.1.134.253:34103: connect: Connection refused (error 111) 04:09:34.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.622771 1650 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 124016us 04:09:34.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:34 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608574, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:34.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.625030 1650 init.cc:359] Logged in from keytab as kudu/127.1.134.253@KRBTEST.COM (short username kudu) 04:09:34.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.628813 1650 fs_manager.cc:263] Metadata directory not provided 04:09:34.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.628939 1650 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:34.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.629168 1650 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:09:34.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.629282 1650 server_base.cc:449] Attempting to create new FS layout instead 04:09:34.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.632290 1650 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data/instance: 04:09:34.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "00360890147648a5af70d3e7251384d7" 04:09:34.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:34 on dist-test-slave-dist-test-slave-fvsv" 04:09:34.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.632975 1650 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/wal/instance: 04:09:34.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "00360890147648a5af70d3e7251384d7" 04:09:34.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:34 on dist-test-slave-dist-test-slave-fvsv" 04:09:34.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.642372 1650 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.009s sys 0.001s 04:09:34.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.643587 1650 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:34.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.644064 1650 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:34.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.651211 1650 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s 04:09:34.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.651336 1650 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/wal 04:09:34.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "00360890147648a5af70d3e7251384d7" 04:09:34.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:34 on dist-test-slave-dist-test-slave-fvsv" 04:09:34.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.651561 1650 fs_report.cc:352] FS layout report 04:09:34.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:34.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/wal 04:09:34.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/wal 04:09:34.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data/data 04:09:34.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:34.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:34.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:34.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:34.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:34.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:34.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:34.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:34.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:34.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:34.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:34.789342 1645 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } attempt: 4 04:09:34.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:34.791874 1645 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.253:34103: Network error: Client connection negotiation failed: client connection to 127.1.134.253:34103: connect: Connection refused (error 111) 04:09:35.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.095278 1645 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } attempt: 5 04:09:35.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.097692 1645 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.253:34103: Network error: Client connection negotiation failed: client connection to 127.1.134.253:34103: connect: Connection refused (error 111) 04:09:35.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.311233 1650 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.635s user 0.209s sys 0.398s 04:09:35.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.311491 1650 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.636s user 0.209s sys 0.398s 04:09:35.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.372581 1650 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:35.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.441890 1650 rpc_server.cc:205] RPC server started. Bound to: 127.1.134.253:34103 04:09:35.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.444267 1650 webserver.cc:374] Webserver started at http://127.1.134.253:40569/ using document root and password file 04:09:35.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.445170 1650 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data/info.pb 04:09:35.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.448274 1712 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:09:35.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.455394 1563 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 1650 04:09:35.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.460057 1712 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } has no permanent_uuid. Determining permanent_uuid... 04:09:35.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.1.134.252@KRBTEST.COM; defaulting to no policy 04:09:35.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:35.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.1.134.252@KRBTEST.COM" created. 04:09:35.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:35 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608574, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.253@KRBTEST.COM for kudu/127.1.134.254@KRBTEST.COM 04:09:35.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.491544 1712 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } has no permanent_uuid. Determining permanent_uuid... 04:09:35.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:35.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.134.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.252.keytab. 04:09:35.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.134.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.252.keytab. 04:09:35.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.134.252 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.252.keytab. 04:09:35.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.1.134.252 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.252.keytab. 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.501052 1563 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/wal 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.134.252:45751 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.134.252 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:35.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/logs 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data/info.pb 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.252.keytab 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.134.252 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --leader_failure_max_missed_heartbeat_periods=10.0 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.134.254:46663,127.1.134.253:34103,127.1.134.252:45751 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:35.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:35 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608574, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.253@KRBTEST.COM for kudu/127.1.134.253@KRBTEST.COM 04:09:35.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.512194 1712 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } has no permanent_uuid. Determining permanent_uuid... 04:09:35.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.513609 1712 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.252:45751: Network error: Client connection negotiation failed: client connection to 127.1.134.252:45751: connect: Connection refused (error 111) 04:09:35.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.532994 1712 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } attempt: 1 04:09:35.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.534417 1712 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.252:45751: Network error: Client connection negotiation failed: client connection to 127.1.134.252:45751: connect: Connection refused (error 111) 04:09:35.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.582720 1712 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } attempt: 2 04:09:35.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.584267 1712 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.252:45751: Network error: Client connection negotiation failed: client connection to 127.1.134.252:45751: connect: Connection refused (error 111) 04:09:35.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.632047 1645 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } attempt: 6 04:09:35.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:35 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.254@KRBTEST.COM for kudu/127.1.134.253@KRBTEST.COM 04:09:35.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.653185 1645 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } has no permanent_uuid. Determining permanent_uuid... 04:09:35.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.654692 1645 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.252:45751: Network error: Client connection negotiation failed: client connection to 127.1.134.252:45751: connect: Connection refused (error 111) 04:09:35.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:35.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.655253 1718 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:35.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.655596 1718 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.1.134.252 04:09:35.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.655901 1718 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:35.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.667824 1718 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:35.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.667995 1718 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:35.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.668128 1718 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:35.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.668341 1718 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:35.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.668532 1718 flags.cc:404] Enabled experimental flag: --principal=kudu/127.1.134.252 04:09:35.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.687027 1645 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } attempt: 1 04:09:35.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.687350 1718 master_main.cc:98] Master server non-default flags: 04:09:35.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --leader_failure_max_missed_heartbeat_periods=10 04:09:35.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data 04:09:35.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/wal 04:09:35.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:35.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.134.254:46663,127.1.134.253:34103,127.1.134.252:45751 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.134.252:45751 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.252.keytab 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.134.252 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data/info.pb 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.134.252 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:35.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/logs 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:35.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.689198 1645 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.252:45751: Network error: Client connection negotiation failed: client connection to 127.1.134.252:45751: connect: Connection refused (error 111) 04:09:35.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.695585 1712 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } attempt: 3 04:09:35.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.697070 1712 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.252:45751: Network error: Client connection negotiation failed: client connection to 127.1.134.252:45751: connect: Connection refused (error 111) 04:09:35.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.702136 1718 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 124516us 04:09:35.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:35 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608575, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:35.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.704495 1718 init.cc:359] Logged in from keytab as kudu/127.1.134.252@KRBTEST.COM (short username kudu) 04:09:35.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.707715 1718 fs_manager.cc:263] Metadata directory not provided 04:09:35.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.707835 1718 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:35.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.708060 1718 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:09:35.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.708163 1718 server_base.cc:449] Attempting to create new FS layout instead 04:09:35.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.711153 1718 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data/instance: 04:09:35.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "d45578f099d04b35bda8327c24407546" 04:09:35.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:35 on dist-test-slave-dist-test-slave-fvsv" 04:09:35.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.711865 1718 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/wal/instance: 04:09:35.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "d45578f099d04b35bda8327c24407546" 04:09:35.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:35 on dist-test-slave-dist-test-slave-fvsv" 04:09:35.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.720630 1718 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s 04:09:35.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.721832 1718 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:35.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.722321 1718 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.729065 1718 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.729190 1718 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/wal 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "d45578f099d04b35bda8327c24407546" 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:35 on dist-test-slave-dist-test-slave-fvsv" 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.729400 1718 fs_report.cc:352] FS layout report 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/wal 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/wal 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data/data 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:35.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:35.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:35.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:35.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:35.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:35.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.757540 1645 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } attempt: 2 04:09:35.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.759181 1645 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.252:45751: Network error: Client connection negotiation failed: client connection to 127.1.134.252:45751: connect: Connection refused (error 111) 04:09:35.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.835479 1645 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } attempt: 3 04:09:35.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.837088 1645 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.252:45751: Network error: Client connection negotiation failed: client connection to 127.1.134.252:45751: connect: Connection refused (error 111) 04:09:35.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.873371 1712 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } attempt: 4 04:09:35.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.874997 1712 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.252:45751: Network error: Client connection negotiation failed: client connection to 127.1.134.252:45751: connect: Connection refused (error 111) 04:09:35.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:35.974398 1645 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } attempt: 4 04:09:35.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:35.976039 1645 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.252:45751: Network error: Client connection negotiation failed: client connection to 127.1.134.252:45751: connect: Connection refused (error 111) 04:09:36.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.178421 1712 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } attempt: 5 04:09:36.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:36.180891 1712 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.252:45751: Network error: Client connection negotiation failed: client connection to 127.1.134.252:45751: connect: Connection refused (error 111) 04:09:36.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.259394 1645 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } attempt: 5 04:09:36.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:36.261696 1645 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.1.134.252:45751: Network error: Client connection negotiation failed: client connection to 127.1.134.252:45751: connect: Connection refused (error 111) 04:09:36.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:36.455319 1718 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.696s user 0.000s sys 0.002s 04:09:36.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:36.455533 1718 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.696s user 0.000s sys 0.002s 04:09:36.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.479470 1718 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:36.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.553762 1718 rpc_server.cc:205] RPC server started. Bound to: 127.1.134.252:45751 04:09:36.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.557067 1718 webserver.cc:374] Webserver started at http://127.1.134.252:41503/ using document root and password file 04:09:36.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.557971 1718 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data/info.pb 04:09:36.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.561676 1781 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:09:36.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.567868 1563 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 1718 04:09:36.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.577363 1781 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } has no permanent_uuid. Determining permanent_uuid... 04:09:36.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608575, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.252@KRBTEST.COM for kudu/127.1.134.254@KRBTEST.COM 04:09:36.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.134.252@KRBTEST.COM 04:09:36.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.613198 1781 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } has no permanent_uuid. Determining permanent_uuid... 04:09:36.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608575, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.252@KRBTEST.COM for kudu/127.1.134.253@KRBTEST.COM 04:09:36.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.134.253@KRBTEST.COM 04:09:36.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.636955 1781 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } has no permanent_uuid. Determining permanent_uuid... 04:09:36.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608575, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.252@KRBTEST.COM for kudu/127.1.134.252@KRBTEST.COM 04:09:36.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.134.254@KRBTEST.COM 04:09:36.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.657346 1563 external_mini_cluster.cc:628] 0 TS(s) registered with all masters 04:09:36.657 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:36.657 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_kdc_env_vars { } 04:09:36.658 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_kdc_env_vars { env_vars { key: "KUDU_ENABLE_KRB5_REALM_FIX" value: "yes" } env_vars { key: "KRB5_KDC_PROFILE" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kdc.conf" } env_vars { key: "KRB5_CONFIG" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5.conf" } env_vars { key: "KRB5CCNAME" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5cc" } } 04:09:36.659 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_masters { } 04:09:36.659 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.1.134.254" port: 46663 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.1.134.253" port: 34103 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.1.134.252" port: 45751 } } } 04:09:36.660 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_tservers { } 04:09:36.660 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:36.661 [INFO - main] (MiniKuduCluster.java:439) Running kinit for user test-admin 04:09:36.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.660981 1781 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: Bootstrap starting. 04:09:36.661 [DEBUG - main] (MiniKuduCluster.java:155) Request: kinit { username: "test-admin" } 04:09:36.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.664604 1781 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: No blocks or log segments found. Creating new log. 04:09:36.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.666385 1781 log.cc:525] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: Log is configured to *not* fsync() on all Append() calls 04:09:36.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.669878 1781 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: No bootstrap required, opened a new log 04:09:36.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:36.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:09:36.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.684032 1781 raft_consensus.cc:340] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:36.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.684706 1781 raft_consensus.cc:366] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:09:36.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.684880 1781 raft_consensus.cc:705] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45578f099d04b35bda8327c24407546, State: Initialized, Role: FOLLOWER 04:09:36.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.685477 1781 consensus_queue.cc:229] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:36.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.687538 1781 sys_catalog.cc:339] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:36.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.687991 1781 sys_catalog.cc:342] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: This master's current role is: FOLLOWER 04:09:36.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.689011 1781 sys_catalog.cc:427] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: configured and running, proceeding with master startup. 04:09:36.691 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:36.699 [DEBUG - main] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5cc 04:09:36.700 [DEBUG - main] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM => started cluster 04:09:36.712 [DEBUG - New I/O worker #39] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.1.134.253 04:09:36.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.715240 1712 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } attempt: 6 04:09:36.720 [DEBUG - New I/O worker #37] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.1.134.254 04:09:36.722 [DEBUG - New I/O worker #38] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.1.134.252 04:09:36.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.134.253@KRBTEST.COM 04:09:36.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.134.254@KRBTEST.COM 04:09:36.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.134.252@KRBTEST.COM 04:09:36.765 [DEBUG - New I/O worker #39] (Negotiator.java:764) Authenticated connection [id: 0xab7906b8, /127.0.0.1:49904 => /127.1.134.253:34103] using SASL/GSSAPI 04:09:36.769 [INFO - New I/O worker #39] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.253:34103: Catalog manager is not initialized. State: Starting 04:09:36.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.774082 1645 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } attempt: 6 04:09:36.789 [DEBUG - New I/O worker #38] (Negotiator.java:764) Authenticated connection [id: 0xbef355d2, /127.0.0.1:37804 => /127.1.134.252:45751] using SASL/GSSAPI 04:09:36.789 [DEBUG - New I/O worker #37] (Negotiator.java:764) Authenticated connection [id: 0xbd8e7477, /127.0.0.1:36626 => /127.1.134.254:46663] using SASL/GSSAPI 04:09:36.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608574, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.253@KRBTEST.COM for kudu/127.1.134.252@KRBTEST.COM 04:09:36.796 [INFO - New I/O worker #37] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.254:46663: Catalog manager is not initialized. State: Starting 04:09:36.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608573, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.254@KRBTEST.COM for kudu/127.1.134.252@KRBTEST.COM 04:09:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.687762 1791 sys_catalog.cc:339] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.844360 1791 sys_catalog.cc:342] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: This master's current role is: FOLLOWER 04:09:36.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:36.735391 1737 tablet.cc:1799] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:09:36.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.891340 1791 raft_consensus.cc:472] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:09:36.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.892298 1791 raft_consensus.cc:494] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:36.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.896914 1645 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: Bootstrap starting. 04:09:36.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.899718 1645 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: No blocks or log segments found. Creating new log. 04:09:36.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.901134 1645 log.cc:525] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: Log is configured to *not* fsync() on all Append() calls 04:09:36.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.903625 1791 leader_election.cc:296] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35094e87862c4278baa1b66918879cc2 (127.1.134.254:46663), 00360890147648a5af70d3e7251384d7 (127.1.134.253:34103) 04:09:36.905 [WARN - New I/O worker #38] (ConnectToCluster.java:281) Unable to find the leader master 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103; will retry 04:09:36.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.907449 1645 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: No bootstrap required, opened a new log 04:09:36.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:36.908160 1792 catalog_manager.cc:1122] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:36.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:36.908352 1792 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:36.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.911463 1712 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7: Bootstrap starting. 04:09:36.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.918913 1712 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7: No blocks or log segments found. Creating new log. 04:09:36.925 [INFO - New I/O worker #37] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.254:46663: Catalog manager is not initialized. State: Starting 04:09:36.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.922976 1712 log.cc:525] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7: Log is configured to *not* fsync() on all Append() calls 04:09:36.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.925060 1645 raft_consensus.cc:340] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:36.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.925822 1645 raft_consensus.cc:366] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:09:36.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.926054 1645 raft_consensus.cc:705] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35094e87862c4278baa1b66918879cc2, State: Initialized, Role: FOLLOWER 04:09:36.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.926868 1645 consensus_queue.cc:229] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:36.931 [INFO - New I/O worker #39] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.253:34103: Catalog manager is not initialized. State: Starting 04:09:36.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.929699 1712 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7: No bootstrap required, opened a new log 04:09:36.933 [WARN - New I/O worker #38] (ConnectToCluster.java:281) Unable to find the leader master 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103; will retry 04:09:36.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.939821 1645 sys_catalog.cc:339] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:36.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.940688 1645 sys_catalog.cc:342] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: This master's current role is: FOLLOWER 04:09:36.944 [INFO - New I/O worker #39] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.253:34103: Catalog manager is not initialized. State: Starting 04:09:36.945 [INFO - New I/O worker #37] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.254:46663: Catalog manager is not initialized. State: Starting 04:09:36.947 [WARN - New I/O worker #38] (ConnectToCluster.java:281) Unable to find the leader master 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103; will retry 04:09:36.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.948154 1645 sys_catalog.cc:427] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: configured and running, proceeding with master startup. 04:09:36.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.961906 1712 raft_consensus.cc:340] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:36.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.962640 1712 raft_consensus.cc:366] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:09:36.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.962875 1712 raft_consensus.cc:705] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00360890147648a5af70d3e7251384d7, State: Initialized, Role: FOLLOWER 04:09:36.965 [INFO - New I/O worker #39] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.253:34103: Catalog manager is not initialized. State: Starting 04:09:36.967 [INFO - New I/O worker #37] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.254:46663: Catalog manager is not initialized. State: Starting 04:09:36.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.968405 1712 consensus_queue.cc:229] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:36.970 [WARN - New I/O worker #38] (ConnectToCluster.java:281) Unable to find the leader master 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103; will retry 04:09:36.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.977005 1812 sys_catalog.cc:339] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:36.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.977713 1812 sys_catalog.cc:342] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: This master's current role is: FOLLOWER 04:09:36.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.978704 1712 sys_catalog.cc:339] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:36.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.979290 1712 sys_catalog.cc:342] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: This master's current role is: FOLLOWER 04:09:36.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.980424 1712 sys_catalog.cc:427] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: configured and running, proceeding with master startup. 04:09:36.987 [INFO - New I/O worker #39] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.253:34103: Catalog manager is not initialized. State: Starting 04:09:36.991 [INFO - New I/O worker #37] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.254:46663: Catalog manager is not initialized. State: Starting 04:09:37.005 [WARN - New I/O worker #38] (ConnectToCluster.java:281) Unable to find the leader master 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103; will retry 04:09:37.027 [INFO - New I/O worker #37] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.254:46663: Catalog manager is not initialized. State: Starting 04:09:37.032 [INFO - New I/O worker #39] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.253:34103: Catalog manager is not initialized. State: Starting 04:09:37.033 [WARN - New I/O worker #39] (ConnectToCluster.java:281) Unable to find the leader master 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103; will retry 04:09:37.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.039343 1622 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d45578f099d04b35bda8327c24407546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35094e87862c4278baa1b66918879cc2" is_pre_election: true 04:09:37.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.040320 1622 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45578f099d04b35bda8327c24407546 in term 0. 04:09:37.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.042569 1730 leader_election.cc:310] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35094e87862c4278baa1b66918879cc2, d45578f099d04b35bda8327c24407546; no voters: 04:09:37.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.940075 1810 sys_catalog.cc:339] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:37.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.043691 1810 sys_catalog.cc:342] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: This master's current role is: FOLLOWER 04:09:37.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.044296 1791 raft_consensus.cc:2644] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 0 FOLLOWER]: Leader pre-election won for term 1 04:09:37.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.044565 1791 raft_consensus.cc:472] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:09:37.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.044809 1791 raft_consensus.cc:2890] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 0 FOLLOWER]: Advancing to term 1 04:09:37.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.050595 1791 raft_consensus.cc:494] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:37.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.052498 1811 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:37.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.052757 1811 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:37.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.052783 1601 tablet.cc:1799] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:09:37.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.052940 1622 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d45578f099d04b35bda8327c24407546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35094e87862c4278baa1b66918879cc2" 04:09:37.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.053376 1622 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 0 FOLLOWER]: Advancing to term 1 04:09:37.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.054652 1791 leader_election.cc:296] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [CANDIDATE]: Term 1 election: Requested vote from peers 35094e87862c4278baa1b66918879cc2 (127.1.134.254:46663), 00360890147648a5af70d3e7251384d7 (127.1.134.253:34103) 04:09:37.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.058199 1622 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45578f099d04b35bda8327c24407546 in term 1. 04:09:37.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.059088 1730 leader_election.cc:310] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35094e87862c4278baa1b66918879cc2, d45578f099d04b35bda8327c24407546; no voters: 04:09:37.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.059777 1791 raft_consensus.cc:2644] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 1 FOLLOWER]: Leader election won for term 1 04:09:37.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.063642 1791 raft_consensus.cc:667] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 1 LEADER]: Becoming Leader. State: Replica: d45578f099d04b35bda8327c24407546, State: Running, Role: LEADER 04:09:37.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.064297 1791 consensus_queue.cc:206] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:37.068 [INFO - New I/O worker #39] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.253:34103: Catalog manager is not initialized. State: Starting 04:09:37.077 [WARN - New I/O worker #38] (ConnectToCluster.java:281) Unable to find the leader master 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103; will retry 04:09:37.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.078588 1815 sys_catalog.cc:339] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d45578f099d04b35bda8327c24407546. Latest consensus state: current_term: 1 leader_uuid: "d45578f099d04b35bda8327c24407546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:37.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.079444 1815 sys_catalog.cc:342] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: This master's current role is: LEADER 04:09:37.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.084251 1817 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:09:37.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.090124 1817 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:09:37.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:36.980634 1689 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d45578f099d04b35bda8327c24407546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00360890147648a5af70d3e7251384d7" is_pre_election: true 04:09:37.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.123323 1689 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45578f099d04b35bda8327c24407546 in term 0. 04:09:37.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.055462 1688 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d45578f099d04b35bda8327c24407546" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00360890147648a5af70d3e7251384d7" 04:09:37.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.127002 1688 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 0 FOLLOWER]: Advancing to term 1 04:09:37.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.129184 1668 tablet.cc:1799] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:09:37.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.134024 1688 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45578f099d04b35bda8327c24407546 in term 1. 04:09:37.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.140924 1813 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:37.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.141150 1813 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:37.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.144100 1622 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 1 FOLLOWER]: Refusing update from remote peer d45578f099d04b35bda8327c24407546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:09:37.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.145460 1688 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 1 FOLLOWER]: Refusing update from remote peer d45578f099d04b35bda8327c24407546: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:09:37.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.145694 1815 consensus_queue.cc:984] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:09:37.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.146982 1815 consensus_queue.cc:984] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:09:37.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.165603 1810 sys_catalog.cc:339] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d45578f099d04b35bda8327c24407546. Latest consensus state: current_term: 1 leader_uuid: "d45578f099d04b35bda8327c24407546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:37.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.166318 1810 sys_catalog.cc:342] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: This master's current role is: FOLLOWER 04:09:37.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.172158 1812 sys_catalog.cc:339] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d45578f099d04b35bda8327c24407546. Latest consensus state: current_term: 1 leader_uuid: "d45578f099d04b35bda8327c24407546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:37.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.172889 1812 sys_catalog.cc:342] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: This master's current role is: FOLLOWER 04:09:37.174 [WARN - New I/O worker #39] (ConnectToCluster.java:278) None of the provided masters 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103 is a leader; will retry 04:09:37.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.176343 1818 mvcc.cc:203] Tried to move safe_time back from 6396348731954429952 to 6396348731663310848. Current Snapshot: MvccSnapshot[committed={T|T < 6396348731954429952}] 04:09:37.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.177932 1815 sys_catalog.cc:339] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d45578f099d04b35bda8327c24407546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:37.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.178591 1815 sys_catalog.cc:342] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: This master's current role is: LEADER 04:09:37.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.180719 1791 sys_catalog.cc:339] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d45578f099d04b35bda8327c24407546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:37.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.181430 1791 sys_catalog.cc:342] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: This master's current role is: LEADER 04:09:37.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.183301 1820 mvcc.cc:203] Tried to move safe_time back from 6396348731954429952 to 6396348731663310848. Current Snapshot: MvccSnapshot[committed={T|T < 6396348731954429952}] 04:09:37.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.187531 1821 mvcc.cc:203] Tried to move safe_time back from 6396348731954429952 to 6396348731663310848. Current Snapshot: MvccSnapshot[committed={T|T < 6396348731954429952}] 04:09:37.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.192044 1812 sys_catalog.cc:339] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d45578f099d04b35bda8327c24407546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:37.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.192744 1812 sys_catalog.cc:342] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: This master's current role is: FOLLOWER 04:09:37.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.195998 1810 sys_catalog.cc:339] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d45578f099d04b35bda8327c24407546" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:37.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.196732 1810 sys_catalog.cc:342] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: This master's current role is: FOLLOWER 04:09:37.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.201550 1817 catalog_manager.cc:959] Generated new certificate authority record 04:09:37.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.203790 1817 catalog_manager.cc:1084] Loading token signing keys... 04:09:37.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.244868 1817 catalog_manager.cc:4350] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: Generated new TSK 0 04:09:37.330 [DEBUG - New I/O worker #38] (AsyncKuduClient.java:2322) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) => exported auth 04:09:37.368 [DEBUG - main] (SecurityUtil.java:121) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 04:09:37.369 [DEBUG - main] (SecurityContext.java:312) Importing authentication credentials with one authn token, 1 cert(s), and realUser=slave => imported auth 04:09:37.369 [INFO - main] (MiniKuduCluster.java:330) Killing master server 127.1.134.254:46663 04:09:37.370 [DEBUG - main] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:09:37.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.370911 1563 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 1583 04:09:37.395 [DEBUG - New I/O worker #37] (Connection.java:691) [peer master-127.1.134.254:46663(127.1.134.254:46663)] cleaning up while in state READY due to: connection disconnected 04:09:37.410 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:37.412 [INFO - main] (MiniKuduCluster.java:330) Killing master server 127.1.134.252:45751 04:09:37.412 [DEBUG - main] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:09:37.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.412744 1563 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 1718 04:09:37.434 [DEBUG - New I/O worker #38] (Connection.java:691) [peer master-127.1.134.252:45751(127.1.134.252:45751)] cleaning up while in state READY due to: connection disconnected 04:09:37.437 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:37.438 [INFO - main] (MiniKuduCluster.java:330) Killing master server 127.1.134.253:34103 04:09:37.438 [DEBUG - main] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:09:37.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.438908 1563 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 1650 04:09:37.455 [DEBUG - New I/O worker #39] (Connection.java:691) [peer master-127.1.134.253:34103(127.1.134.253:34103)] cleaning up while in state READY due to: connection disconnected 04:09:37.455 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:37.456 [INFO - main] (MiniKuduCluster.java:311) Starting master server 127.1.134.254:46663 04:09:37.456 [DEBUG - main] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:09:37.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.457435 1563 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:37.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:37.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/wal 04:09:37.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.134.254:46663 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.134.254 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=40647 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/logs 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data/info.pb 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:37.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:37.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.254.keytab 04:09:37.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.134.254 04:09:37.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:37.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:37.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:37.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --leader_failure_max_missed_heartbeat_periods=10.0 04:09:37.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:37.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.134.254:46663,127.1.134.253:34103,127.1.134.252:45751 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:37.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:37.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.604472 1836 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:37.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.604837 1836 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.1.134.254 04:09:37.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.605093 1836 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:37.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.616797 1836 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:37.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.616953 1836 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:37.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.617051 1836 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:37.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.617157 1836 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:37.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:37.617274 1836 flags.cc:404] Enabled experimental flag: --principal=kudu/127.1.134.254 04:09:37.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.635468 1836 master_main.cc:98] Master server non-default flags: 04:09:37.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --leader_failure_max_missed_heartbeat_periods=10 04:09:37.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data 04:09:37.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/wal 04:09:37.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.134.254:46663,127.1.134.253:34103,127.1.134.252:45751 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.134.254:46663 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.254.keytab 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.134.254 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data/info.pb 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.134.254 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=40647 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/logs 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:37.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:37.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:37.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:37.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.649363 1836 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 125516us 04:09:37.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:37 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608577, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:37.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.651661 1836 init.cc:359] Logged in from keytab as kudu/127.1.134.254@KRBTEST.COM (short username kudu) 04:09:37.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.654299 1836 fs_manager.cc:263] Metadata directory not provided 04:09:37.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.654450 1836 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:37.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.660066 1836 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s 04:09:37.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.660761 1836 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:37.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.661309 1836 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:37.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.668543 1836 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.668680 1836 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/wal 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "35094e87862c4278baa1b66918879cc2" 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:33 on dist-test-slave-dist-test-slave-fvsv" 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:37.670008 1836 fs_report.cc:352] FS layout report 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/wal 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/wal 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data/data 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:37.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:37.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:37.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:37.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:37.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:38.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:38.532780 1836 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.834s user 0.000s sys 0.002s 04:09:38.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:38.533056 1836 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.834s user 0.000s sys 0.002s 04:09:38.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.629820 1836 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:38.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.685292 1836 rpc_server.cc:205] RPC server started. Bound to: 127.1.134.254:46663 04:09:38.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.687672 1836 webserver.cc:374] Webserver started at http://127.1.134.254:40647/ using document root and password file 04:09:38.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.688649 1836 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-0/data/info.pb 04:09:38.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.694458 1563 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 1836 04:09:38.694 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:38.695 [INFO - main] (MiniKuduCluster.java:311) Starting master server 127.1.134.252:45751 04:09:38.695 [DEBUG - main] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:09:38.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.695960 1898 sys_catalog.cc:188] Verifying existing consensus state 04:09:38.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.696241 1563 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:38.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:38.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/wal 04:09:38.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data 04:09:38.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:38.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:38.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:38.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:38.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.134.252:45751 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.134.252 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=41503 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/logs 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data/info.pb 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:38.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.252.keytab 04:09:38.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.134.252 04:09:38.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:38.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:38.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:38.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --leader_failure_max_missed_heartbeat_periods=10.0 04:09:38.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:38.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.134.254:46663,127.1.134.253:34103,127.1.134.252:45751 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:38.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.700691 1898 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: Bootstrap starting. 04:09:38.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.728080 1898 log.cc:525] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: Log is configured to *not* fsync() on all Append() calls 04:09:38.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.738281 1898 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:09:38.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.738941 1898 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: Bootstrap complete. 04:09:38.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.758448 1898 raft_consensus.cc:340] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:38.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.763599 1898 raft_consensus.cc:705] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35094e87862c4278baa1b66918879cc2, State: Initialized, Role: FOLLOWER 04:09:38.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.764382 1898 consensus_queue.cc:229] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:38.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.766379 1898 sys_catalog.cc:339] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.766845 1898 sys_catalog.cc:342] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: This master's current role is: FOLLOWER 04:09:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.768199 1898 sys_catalog.cc:427] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: configured and running, proceeding with master startup. 04:09:38.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.766861 1901 sys_catalog.cc:339] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:38.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.837854 1901 sys_catalog.cc:342] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: This master's current role is: FOLLOWER 04:09:38.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.849093 1902 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: acquiring CA information for follower catalog manager: success 04:09:38.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.852020 1902 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:09:38.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:38.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:38.944753 1899 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:38.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:38.945103 1899 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.1.134.252 04:09:38.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:38.945329 1899 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:38.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:38.957515 1899 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:38.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:38.957680 1899 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:38.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:38.957774 1899 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:38.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:38.957887 1899 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:38.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:38.958005 1899 flags.cc:404] Enabled experimental flag: --principal=kudu/127.1.134.252 04:09:38.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.979499 1899 master_main.cc:98] Master server non-default flags: 04:09:38.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --leader_failure_max_missed_heartbeat_periods=10 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/wal 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.134.254:46663,127.1.134.253:34103,127.1.134.252:45751 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.134.252:45751 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.252.keytab 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.134.252 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data/info.pb 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.134.252 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=41503 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/logs 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.993772 1899 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 126016us 04:09:38.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:38 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608578, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:38.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:38.996024 1899 init.cc:359] Logged in from keytab as kudu/127.1.134.252@KRBTEST.COM (short username kudu) 04:09:39.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:39.007777 1899 fs_manager.cc:263] Metadata directory not provided 04:09:39.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:39.007930 1899 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:39.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:39.012760 1899 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s 04:09:39.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:39.013396 1899 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:39.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:39.013895 1899 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:39.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:39.032383 1899 fs_manager.cc:419] Time spent opening block manager: real 0.014s user 0.002s sys 0.011s 04:09:39.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:39.032531 1899 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/wal 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "d45578f099d04b35bda8327c24407546" 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:35 on dist-test-slave-dist-test-slave-fvsv" 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:39.033926 1899 fs_report.cc:352] FS layout report 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/wal 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/wal 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data/data 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:39.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:39.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:39.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:39.927372 1899 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.863s user 0.287s sys 0.536s 04:09:39.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:39.927593 1899 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.864s user 0.287s sys 0.536s 04:09:40.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.008566 1899 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:40.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.077464 1899 rpc_server.cc:205] RPC server started. Bound to: 127.1.134.252:45751 04:09:40.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.082958 1899 webserver.cc:374] Webserver started at http://127.1.134.252:41503/ using document root and password file 04:09:40.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.084184 1899 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-2/data/info.pb 04:09:40.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.088506 1563 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 1899 04:09:40.088 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:40.089 [INFO - main] (MiniKuduCluster.java:311) Starting master server 127.1.134.253:34103 04:09:40.089 [DEBUG - main] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:09:40.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.090456 1563 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:40.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:40.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/wal 04:09:40.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data 04:09:40.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:40.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:40.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:40.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:40.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.134.253:34103 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.134.253 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=40569 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/logs 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data/info.pb 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.253.keytab 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.134.253 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --leader_failure_max_missed_heartbeat_periods=10.0 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:40.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.134.254:46663,127.1.134.253:34103,127.1.134.252:45751 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:40.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.092490 1965 sys_catalog.cc:188] Verifying existing consensus state 04:09:40.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.097394 1965 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: Bootstrap starting. 04:09:40.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.125679 1965 log.cc:525] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: Log is configured to *not* fsync() on all Append() calls 04:09:40.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.134732 1965 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:09:40.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.135354 1965 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: Bootstrap complete. 04:09:40.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.155238 1965 raft_consensus.cc:340] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:40.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.160333 1965 raft_consensus.cc:705] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45578f099d04b35bda8327c24407546, State: Initialized, Role: FOLLOWER 04:09:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.161015 1965 consensus_queue.cc:229] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:40.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.163565 1965 sys_catalog.cc:339] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:40.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.164155 1965 sys_catalog.cc:342] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: This master's current role is: FOLLOWER 04:09:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.165223 1965 sys_catalog.cc:427] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: configured and running, proceeding with master startup. 04:09:40.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.164057 1968 sys_catalog.cc:339] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:40.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.234716 1968 sys_catalog.cc:342] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: This master's current role is: FOLLOWER 04:09:40.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.245244 1969 catalog_manager.cc:1120] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: acquiring CA information for follower catalog manager: success 04:09:40.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.248380 1969 catalog_manager.cc:1148] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:09:40.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:40.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:40.280625 1966 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:40.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:40.280949 1966 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.1.134.253 04:09:40.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:40.281208 1966 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:40.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:40.292222 1966 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:40.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:40.292383 1966 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:40.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:40.292476 1966 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:40.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:40.292583 1966 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:40.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:40.292701 1966 flags.cc:404] Enabled experimental flag: --principal=kudu/127.1.134.253 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.311861 1966 master_main.cc:98] Master server non-default flags: 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --leader_failure_max_missed_heartbeat_periods=10 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/wal 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.1.134.254:46663,127.1.134.253:34103,127.1.134.252:45751 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.1.134.253:34103 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/krb5kdc/kudu_127.1.134.253.keytab 04:09:40.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.1.134.253 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data/info.pb 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.1.134.253 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=40569 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/logs 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:40.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:40.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:40.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:40.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:40.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:40.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:40.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:40.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.325996 1966 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 127016us 04:09:40.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:40 dist-test-slave-dist-test-slave-fvsv krb5kdc[1572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608580, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.134.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:40.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.328270 1966 init.cc:359] Logged in from keytab as kudu/127.1.134.253@KRBTEST.COM (short username kudu) 04:09:40.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.343319 1966 fs_manager.cc:263] Metadata directory not provided 04:09:40.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.343470 1966 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:40.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.348354 1966 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s 04:09:40.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.348987 1966 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:40.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.349500 1966 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:40.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.368412 1966 fs_manager.cc:419] Time spent opening block manager: real 0.014s user 0.003s sys 0.010s 04:09:40.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.368554 1966 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/wal 04:09:40.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "00360890147648a5af70d3e7251384d7" 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:34 on dist-test-slave-dist-test-slave-fvsv" 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:40.369853 1966 fs_report.cc:352] FS layout report 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/wal 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/wal 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data/data 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:40.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:41.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:41.214939 1966 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.812s user 0.001s sys 0.001s 04:09:41.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:41.215198 1966 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.812s user 0.002s sys 0.001s 04:09:41.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.246423 1966 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:41.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.367370 1966 rpc_server.cc:205] RPC server started. Bound to: 127.1.134.253:34103 04:09:41.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.370255 1966 webserver.cc:374] Webserver started at http://127.1.134.253:40569/ using document root and password file 04:09:41.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.371220 1966 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster5242194380584136654/master-1/data/info.pb 04:09:41.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.374745 1563 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 1966 04:09:41.375 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:09:41.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.378707 2032 sys_catalog.cc:188] Verifying existing consensus state 04:09:41.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.387148 2032 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7: Bootstrap starting. 04:09:41.403 [DEBUG - New I/O worker #48] (Connection.java:691) [peer master-127.1.134.253:34103(127.1.134.253:34103)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:09:41.403 [INFO - New I/O worker #48] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.253:34103: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:09:41.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:41.406432 2036 negotiation.cc:313] Failed RPC negotiation. Trace: 04:09:41.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.382501 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:49914 04:09:41.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.387661 (+ 5160us) server_negotiation.cc:184] Beginning negotiation 04:09:41.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.387693 (+ 32us) server_negotiation.cc:373] Waiting for connection header 04:09:41.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.387759 (+ 66us) server_negotiation.cc:381] Connection header received 04:09:41.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.401101 (+ 13342us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:09:41.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.401144 (+ 43us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:09:41.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.401461 (+ 317us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:09:41.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.404492 (+ 3031us) server_negotiation.cc:205] Negotiated authn=SASL 04:09:41.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.405195 (+ 703us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49914: BlockingRecv error: recv got EOF from 127.0.0.1:49914 (error 108) 04:09:41.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":4755,"thread_start_us":1704,"threads_started":1} 04:09:41.487 [DEBUG - New I/O worker #47] (Negotiator.java:764) Authenticated connection [id: 0xb432034a, /127.0.0.1:37814 => /127.1.134.252:45751] using TOKEN/null 04:09:41.502 [DEBUG - New I/O worker #46] (Negotiator.java:764) Authenticated connection [id: 0x9169e973, /127.0.0.1:36636 => /127.1.134.254:46663] using TOKEN/null 04:09:41.506 [WARN - New I/O worker #46] (ConnectToCluster.java:281) Unable to find the leader master 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103; will retry 04:09:41.524 [DEBUG - New I/O worker #49] (Connection.java:691) [peer master-127.1.134.253:34103(127.1.134.253:34103)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:09:41.524 [INFO - New I/O worker #49] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.253:34103: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:09:41.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:41.527696 2036 negotiation.cc:313] Failed RPC negotiation. Trace: 04:09:41.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.522825 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:49916 04:09:41.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.523268 (+ 443us) server_negotiation.cc:184] Beginning negotiation 04:09:41.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.523290 (+ 22us) server_negotiation.cc:373] Waiting for connection header 04:09:41.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.523329 (+ 39us) server_negotiation.cc:381] Connection header received 04:09:41.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.523451 (+ 122us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:09:41.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.523471 (+ 20us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:09:41.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.523602 (+ 131us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:09:41.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.523764 (+ 162us) server_negotiation.cc:205] Negotiated authn=SASL 04:09:41.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.526529 (+ 2765us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49916: BlockingRecv error: recv got EOF from 127.0.0.1:49916 (error 108) 04:09:41.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":139} 04:09:41.528 [WARN - New I/O worker #46] (ConnectToCluster.java:281) Unable to find the leader master 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103; will retry 04:09:41.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.439920 2032 log.cc:525] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7: Log is configured to *not* fsync() on all Append() calls 04:09:41.543 [DEBUG - New I/O worker #50] (Connection.java:691) [peer master-127.1.134.253:34103(127.1.134.253:34103)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:09:41.543 [INFO - New I/O worker #50] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.253:34103: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:09:41.544 [WARN - New I/O worker #46] (ConnectToCluster.java:281) Unable to find the leader master 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103; will retry 04:09:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:41.544723 2036 negotiation.cc:313] Failed RPC negotiation. Trace: 04:09:41.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.541432 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:49918 04:09:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.541897 (+ 465us) server_negotiation.cc:184] Beginning negotiation 04:09:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.541919 (+ 22us) server_negotiation.cc:373] Waiting for connection header 04:09:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.541962 (+ 43us) server_negotiation.cc:381] Connection header received 04:09:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.542089 (+ 127us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:09:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.542112 (+ 23us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:09:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.542256 (+ 144us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:09:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.542413 (+ 157us) server_negotiation.cc:205] Negotiated authn=SASL 04:09:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.544005 (+ 1592us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49918: BlockingRecv error: recv got EOF from 127.0.0.1:49918 (error 108) 04:09:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":123} 04:09:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.544894 2032 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:09:41.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.545629 2032 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7: Bootstrap complete. 04:09:41.563 [DEBUG - New I/O worker #51] (Connection.java:691) [peer master-127.1.134.253:34103(127.1.134.253:34103)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:09:41.563 [INFO - New I/O worker #51] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.253:34103: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:09:41.564 [WARN - New I/O worker #47] (ConnectToCluster.java:281) Unable to find the leader master 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103; will retry 04:09:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:41.565084 2036 negotiation.cc:313] Failed RPC negotiation. Trace: 04:09:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.561686 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:49920 04:09:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.562122 (+ 436us) server_negotiation.cc:184] Beginning negotiation 04:09:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.562152 (+ 30us) server_negotiation.cc:373] Waiting for connection header 04:09:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.562193 (+ 41us) server_negotiation.cc:381] Connection header received 04:09:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.562313 (+ 120us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:09:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.562335 (+ 22us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:09:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.562473 (+ 138us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:09:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.563935 (+ 1462us) server_negotiation.cc:205] Negotiated authn=SASL 04:09:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.564390 (+ 455us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49920: BlockingRecv error: recv got EOF from 127.0.0.1:49920 (error 108) 04:09:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":141} 04:09:41.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.558140 2032 raft_consensus.cc:340] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:41.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.565974 2032 raft_consensus.cc:705] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00360890147648a5af70d3e7251384d7, State: Initialized, Role: FOLLOWER 04:09:41.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.566504 2032 consensus_queue.cc:229] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:41.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.568398 2032 sys_catalog.cc:339] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:41.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.568954 2032 sys_catalog.cc:342] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: This master's current role is: FOLLOWER 04:09:41.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.568671 2038 sys_catalog.cc:339] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:41.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.569360 2038 sys_catalog.cc:342] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: This master's current role is: FOLLOWER 04:09:41.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.569974 2032 sys_catalog.cc:427] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: configured and running, proceeding with master startup. 04:09:41.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.584746 2039 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7: acquiring CA information for follower catalog manager: success 04:09:41.585 [DEBUG - New I/O worker #52] (Connection.java:691) [peer master-127.1.134.253:34103(127.1.134.253:34103)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:09:41.585 [INFO - New I/O worker #52] (ConnectToCluster.java:361) Unable to connect to master 127.1.134.253:34103: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:09:41.585 [WARN - New I/O worker #52] (ConnectToCluster.java:281) Unable to find the leader master 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103; will retry 04:09:41.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:41.587416 2036 negotiation.cc:313] Failed RPC negotiation. Trace: 04:09:41.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.581945 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:49922 04:09:41.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.584351 (+ 2406us) server_negotiation.cc:184] Beginning negotiation 04:09:41.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.584368 (+ 17us) server_negotiation.cc:373] Waiting for connection header 04:09:41.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.584397 (+ 29us) server_negotiation.cc:381] Connection header received 04:09:41.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.584491 (+ 94us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:09:41.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.584505 (+ 14us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:09:41.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.584775 (+ 270us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:09:41.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.584902 (+ 127us) server_negotiation.cc:205] Negotiated authn=SASL 04:09:41.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:09:41.586696 (+ 1794us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49922: BlockingRecv error: recv got EOF from 127.0.0.1:49922 (error 108) 04:09:41.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":2191} 04:09:41.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:41.591439 2039 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:09:41.616 [DEBUG - New I/O worker #53] (Negotiator.java:764) Authenticated connection [id: 0x48d6f732, /127.0.0.1:49924 => /127.1.134.253:34103] using TOKEN/null 04:09:41.619 [WARN - New I/O worker #53] (ConnectToCluster.java:278) None of the provided masters 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103 is a leader; will retry 04:09:41.664 [WARN - New I/O worker #53] (ConnectToCluster.java:278) None of the provided masters 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103 is a leader; will retry 04:09:41.784 [WARN - New I/O worker #47] (ConnectToCluster.java:278) None of the provided masters 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103 is a leader; will retry 04:09:41.904 [WARN - New I/O worker #47] (ConnectToCluster.java:278) None of the provided masters 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103 is a leader; will retry 04:09:42.244 [WARN - New I/O worker #53] (ConnectToCluster.java:278) None of the provided masters 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103 is a leader; will retry 04:09:42.667 [WARN - New I/O worker #53] (ConnectToCluster.java:278) None of the provided masters 127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103 is a leader; will retry 04:09:43.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.131968 2042 raft_consensus.cc:472] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:09:43.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.132419 2042 raft_consensus.cc:494] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:43.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.134178 2042 leader_election.cc:296] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00360890147648a5af70d3e7251384d7 (127.1.134.253:34103), d45578f099d04b35bda8327c24407546 (127.1.134.252:45751) 04:09:43.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.149119 2009 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35094e87862c4278baa1b66918879cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "00360890147648a5af70d3e7251384d7" is_pre_election: true 04:09:43.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.149919 2009 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35094e87862c4278baa1b66918879cc2 in term 1. 04:09:43.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.150087 1942 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35094e87862c4278baa1b66918879cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d45578f099d04b35bda8327c24407546" is_pre_election: true 04:09:43.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.150764 1942 raft_consensus.cc:2350] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35094e87862c4278baa1b66918879cc2 in term 1. 04:09:43.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.150843 1846 leader_election.cc:310] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00360890147648a5af70d3e7251384d7, 35094e87862c4278baa1b66918879cc2; no voters: 04:09:43.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.151525 2042 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 1 FOLLOWER]: Leader pre-election won for term 2 04:09:43.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.151700 2042 raft_consensus.cc:472] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:09:43.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.151842 2042 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 1 FOLLOWER]: Advancing to term 2 04:09:43.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.156962 2042 raft_consensus.cc:494] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:43.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.157939 2042 leader_election.cc:296] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [CANDIDATE]: Term 2 election: Requested vote from peers 00360890147648a5af70d3e7251384d7 (127.1.134.253:34103), d45578f099d04b35bda8327c24407546 (127.1.134.252:45751) 04:09:43.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.158625 2009 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35094e87862c4278baa1b66918879cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "00360890147648a5af70d3e7251384d7" 04:09:43.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.158665 1942 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35094e87862c4278baa1b66918879cc2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d45578f099d04b35bda8327c24407546" 04:09:43.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.159113 2009 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 1 FOLLOWER]: Advancing to term 2 04:09:43.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.159152 1942 raft_consensus.cc:2890] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 1 FOLLOWER]: Advancing to term 2 04:09:43.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.166003 2009 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35094e87862c4278baa1b66918879cc2 in term 2. 04:09:43.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.166112 1942 raft_consensus.cc:2350] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35094e87862c4278baa1b66918879cc2 in term 2. 04:09:43.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.166796 1846 leader_election.cc:310] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00360890147648a5af70d3e7251384d7, 35094e87862c4278baa1b66918879cc2; no voters: 04:09:43.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.167418 2042 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 2 FOLLOWER]: Leader election won for term 2 04:09:43.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.168527 2042 raft_consensus.cc:667] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [term 2 LEADER]: Becoming Leader. State: Replica: 35094e87862c4278baa1b66918879cc2, State: Running, Role: LEADER 04:09:43.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.168925 2042 consensus_queue.cc:206] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } 04:09:43.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.172946 2047 sys_catalog.cc:339] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35094e87862c4278baa1b66918879cc2. Latest consensus state: current_term: 2 leader_uuid: "35094e87862c4278baa1b66918879cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:43.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.173581 2047 sys_catalog.cc:342] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: This master's current role is: LEADER 04:09:43.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.176209 2049 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:09:43.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.178704 2049 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:09:43.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.180449 2049 catalog_manager.cc:1084] Loading token signing keys... 04:09:43.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.181620 2049 catalog_manager.cc:4361] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2: Loaded TSK: 0 04:09:43.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.667186 1942 raft_consensus.cc:1184] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [term 2 FOLLOWER]: Refusing update from remote peer 35094e87862c4278baa1b66918879cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) 04:09:43.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.668311 2047 consensus_queue.cc:984] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s 04:09:43.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.676128 2050 sys_catalog.cc:339] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35094e87862c4278baa1b66918879cc2. Latest consensus state: current_term: 2 leader_uuid: "35094e87862c4278baa1b66918879cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:43.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.676862 2050 sys_catalog.cc:342] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: This master's current role is: FOLLOWER 04:09:43.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.680938 2009 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [term 2 FOLLOWER]: Refusing update from remote peer 35094e87862c4278baa1b66918879cc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) 04:09:43.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.680811 2047 sys_catalog.cc:339] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "35094e87862c4278baa1b66918879cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:43.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.681511 2047 sys_catalog.cc:342] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: This master's current role is: LEADER 04:09:43.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.682548 2047 consensus_queue.cc:984] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s 04:09:43.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.683229 2050 sys_catalog.cc:339] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "35094e87862c4278baa1b66918879cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:43.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.683743 2050 sys_catalog.cc:342] T 00000000000000000000000000000000 P d45578f099d04b35bda8327c24407546 [sys.catalog]: This master's current role is: FOLLOWER 04:09:43.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.693303 2059 sys_catalog.cc:339] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "35094e87862c4278baa1b66918879cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.693961 2059 sys_catalog.cc:342] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: This master's current role is: FOLLOWER 04:09:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.693756 2056 sys_catalog.cc:339] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35094e87862c4278baa1b66918879cc2. Latest consensus state: current_term: 2 leader_uuid: "35094e87862c4278baa1b66918879cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:43.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:43.694329 2056 sys_catalog.cc:342] T 00000000000000000000000000000000 P 00360890147648a5af70d3e7251384d7 [sys.catalog]: This master's current role is: FOLLOWER 04:09:44.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:44.263945 2060 sys_catalog.cc:339] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "35094e87862c4278baa1b66918879cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35094e87862c4278baa1b66918879cc2" member_type: VOTER last_known_addr { host: "127.1.134.254" port: 46663 } } peers { permanent_uuid: "00360890147648a5af70d3e7251384d7" member_type: VOTER last_known_addr { host: "127.1.134.253" port: 34103 } } peers { permanent_uuid: "d45578f099d04b35bda8327c24407546" member_type: VOTER last_known_addr { host: "127.1.134.252" port: 45751 } } } 04:09:44.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:44.264541 2060 sys_catalog.cc:342] T 00000000000000000000000000000000 P 35094e87862c4278baa1b66918879cc2 [sys.catalog]: This master's current role is: LEADER 04:09:44.543 [DEBUG - New I/O worker #46] (AsyncKuduClient.java:2322) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 04:09:44.543 [DEBUG - New I/O worker #46] (AsyncKuduClient.java:1296) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=12, TimeoutTracker(timeout=30000, elapsed=3168), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.254:46663, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.252:45751, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.253:34103, [24ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.253:34103: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication, [113ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.252:45751: OK, [127ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.254:46663: OK, [128ms] delaying RPC due to: Service unavailable: Master config (127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication, [141ms] refreshing cache from master, [141ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.254:46663, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.252:45751, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.253:34103, [145ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.253:34103: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication, [146ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.252:45751: OK, [149ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.254:46663: OK, [149ms] delaying RPC due to: Service unavailable: Master config (127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication, [161ms] refreshing cache from master, [161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.254:46663, [161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.252:45751, [162ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.253:34103, [164ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.253:34103: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication, [164ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.252:45751: OK, [165ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.254:46663: OK, [166ms] delaying RPC due to: Service unavailable: Master config (127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication, [181ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.254:46663, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.252:45751, [182ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.253:34103, [184ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.254:46663: OK, [184ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.253:34103: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication, [185ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.252:45751: OK, [185ms] delaying RPC due to: Service unavailable: Master config (127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication, [200ms] refreshing cache from master, [200ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.254:46663, [200ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.252:45751, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.253:34103, [203ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.254:46663: OK, [203ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.252:45751: OK, [206ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.253:34103: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication, [207ms] delaying RPC due to: Service unavailable: Master config (127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication, [220ms] refreshing cache from master, [220ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.254:46663, [220ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.252:45751, [221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.253:34103, [223ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.254:46663: OK, [224ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.252:45751: OK, [240ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.253:34103: OK, [240ms] delaying RPC due to: Service unavailable: Master config (127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103) has no leader., [280ms] refreshing cache from master, [281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.254:46663, [281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.252:45751, [281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.253:34103, [284ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.252:45751: OK, [284ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.254:46663: OK, [285ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.253:34103: OK, [285ms] delaying RPC due to: Service unavailable: Master config (127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103) has no leader., [401ms] refreshing cache from master, [401ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.254:46663, [401ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.252:45751, [402ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.253:34103, [404ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.254:46663: OK, [404ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.253:34103: OK, [405ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.252:45751: OK, [406ms] delaying RPC due to: Service unavailable: Master config (127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103) has no leader., [520ms] refreshing cache from master, [520ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.254:46663, [521ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.252:45751, [521ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.253:34103, [524ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.253:34103: OK, [524ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.254:46663: OK, [524ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.252:45751: OK, [525ms] delaying RPC due to: Service unavailable: Master config (127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103) has no leader., [860ms] refreshing cache from master, [860ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.254:46663, [861ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.252:45751, [861ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.253:34103, [864ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.252:45751: OK, [865ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.254:46663: OK, [865ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.253:34103: OK, [866ms] delaying RPC due to: Service unavailable: Master config (127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103) has no leader., [1281ms] refreshing cache from master, [1281ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.254:46663, [1282ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.252:45751, [1282ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.253:34103, [1284ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.252:45751: OK, [1285ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.254:46663: OK, [1288ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.253:34103: OK, [1288ms] delaying RPC due to: Service unavailable: Master config (127.1.134.254:46663,127.1.134.252:45751,127.1.134.253:34103) has no leader., [3161ms] refreshing cache from master, [3161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.254:46663, [3161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.252:45751, [3161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.1.134.253:34103, [3163ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.254:46663: OK, [3163ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.253:34103: OK, [3164ms] Sub RPC ConnectToMaster: received response from server master-127.1.134.252:45751: OK, deferred=Deferred@884189699(state=PENDING, result=null, callback=(continuation of Deferred@1303373702 after retry RPC after error@1301551725) -> (continuation of Deferred@465620352 after retry RPC after error@847945545) -> (continuation of Deferred@2870601 after retry RPC after error@1837230163) -> (continuation of Deferred@1882227886 after retry RPC after error@379989302) -> (continuation of Deferred@1211759406 after retry RPC after error@714621194) -> (continuation of Deferred@1585030403 after retry RPC after error@2031217767) -> (continuation of Deferred@2121697504 after retry RPC after error@370021941) -> (continuation of Deferred@88930378 after retry RPC after error@1942511184) -> (continuation of Deferred@968096065 after retry RPC after error@1600843772) -> (continuation of Deferred@1032246003 after retry RPC after error@202187213) -> (continuation of Deferred@710060284 after retry RPC after error@584038213), errback=(continuation of Deferred@1303373702 after retry RPC after error@1301551725) -> (continuation of Deferred@465620352 after retry RPC after error@847945545) -> (continuation of Deferred@2870601 after retry RPC after error@1837230163) -> (continuation of Deferred@1882227886 after retry RPC after error@379989302) -> (continuation of Deferred@1211759406 after retry RPC after error@714621194) -> (continuation of Deferred@1585030403 after retry RPC after error@2031217767) -> (continuation of Deferred@2121697504 after retry RPC after error@370021941) -> (continuation of Deferred@88930378 after retry RPC after error@1942511184) -> (continuation of Deferred@968096065 after retry RPC after error@1600843772) -> (continuation of Deferred@1032246003 after retry RPC after error@202187213) -> (continuation of Deferred@710060284 after retry RPC after error@584038213))) after lookup => listTabletServers 04:09:44.552 [DEBUG - New I/O worker #46] (Connection.java:691) [peer master-127.1.134.254:46663(127.1.134.254:46663)] cleaning up while in state READY due to: connection disconnected 04:09:44.552 [DEBUG - New I/O worker #47] (Connection.java:691) [peer master-127.1.134.252:45751(127.1.134.252:45751)] cleaning up while in state READY due to: connection disconnected 04:09:44.552 [DEBUG - New I/O worker #53] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:09:44.553 [DEBUG - New I/O worker #53] (Connection.java:691) [peer master-127.1.134.253:34103(127.1.134.253:34103)] cleaning up while in state READY due to: connection disconnected 04:09:44.553 [DEBUG - main] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:09:44.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:44.554615 1563 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 1836 04:09:44.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:44.574908 1563 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 1966 04:09:44.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:44.592365 1563 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 1899 04:09:45.637 [INFO - main] (RetryRule.java:136) Creating RetryStatement with result reporter and retry count of 0 (no retries) .04:09:45.638 [INFO - Time-limited test] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskusvpEt/test-tmp 04:09:45.638 [INFO - Time-limited test] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin 04:09:45.638 [INFO - Time-limited test] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskusvpEt/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb] 04:09:45.639 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 0 enable_kerberos: true cluster_root: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841" extra_master_flags: "--authn_token_validity_seconds=10" hms_mode: NONE mini_kdc_options { ticket_lifetime: "10s" renew_lifetime: "20s" } } 04:09:45.814 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:45.814 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_cluster { } 04:09:45.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Loading random data 04:09:45.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Initializing database '/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/principal' for realm 'KRBTEST.COM', 04:09:45.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) master key name 'K/M@KRBTEST.COM' 04:09:45.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:45 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): setting up network... 04:09:45.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:45 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) 04:09:45.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked 04:09:45.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 04:09:45.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:45 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): skipping unrecognized local address family 17 04:09:45.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:45 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): skipping unrecognized local address family 17 04:09:45.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:45 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): set up 1 sockets 04:09:45.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:45 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): commencing operation 04:09:45.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: starting... 04:09:46.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:46.403167 2068 mini_kdc.cc:119] Time spent starting KDC: real 0.563s user 0.000s sys 0.016s 04:09:46.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy 04:09:46.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:09:46.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-admin@KRBTEST.COM" created. 04:09:46.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy 04:09:46.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:09:46.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-user@KRBTEST.COM" created. 04:09:46.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy 04:09:46.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:09:46.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "joe-interloper@KRBTEST.COM" created. 04:09:46.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:09:46.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/test-user.keytab. 04:09:46.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/test-user.keytab. 04:09:46.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/test-user.keytab. 04:09:46.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/test-user.keytab. 04:09:46.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:46 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:46.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:09:46.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.2.5.62@KRBTEST.COM; defaulting to no policy 04:09:46.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:46.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.2.5.62@KRBTEST.COM" created. 04:09:46.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:46.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.2.5.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab. 04:09:46.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.2.5.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab. 04:09:46.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.2.5.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab. 04:09:46.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.2.5.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab. 04:09:46.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.554770 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:46.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:46.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:46.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:46.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:46.701336 2088 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:46.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:46.701655 2088 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.62 04:09:46.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:46.701892 2088 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:46.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:46.712739 2088 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:09:46.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:46.712879 2088 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:46.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:46.712985 2088 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:46.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:46.713071 2088 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:46.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:46.713181 2088 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:46.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:46.713304 2088 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.62 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.732064 2088 master_main.cc:98] Master server non-default flags: 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:46.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:46.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:46.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.746244 2088 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 130016us 04:09:46.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:46 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:46.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.748482 2088 init.cc:359] Logged in from keytab as kudu/127.2.5.62@KRBTEST.COM (short username kudu) 04:09:46.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.751785 2088 fs_manager.cc:263] Metadata directory not provided 04:09:46.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.751916 2088 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:46.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.752151 2088 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:09:46.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.752259 2088 server_base.cc:449] Attempting to create new FS layout instead 04:09:46.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.755187 2088 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/instance: 04:09:46.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:09:46.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:46 on dist-test-slave-dist-test-slave-fvsv" 04:09:46.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.755903 2088 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal/instance: 04:09:46.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:09:46.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:46 on dist-test-slave-dist-test-slave-fvsv" 04:09:46.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.764807 2088 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s 04:09:46.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.765986 2088 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:46.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.766479 2088 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:46.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.773201 2088 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s 04:09:46.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.773322 2088 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:46.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:09:46.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:46 on dist-test-slave-dist-test-slave-fvsv" 04:09:46.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:46.773524 2088 fs_report.cc:352] FS layout report 04:09:46.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:46.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:46.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:46.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/data 04:09:46.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:46.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:46.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:46.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:46.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:46.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:46.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:46.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:46.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:46.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:47.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.655988 2088 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.859s user 0.262s sys 0.572s 04:09:47.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.656246 2088 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.859s user 0.262s sys 0.572s 04:09:47.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:47.703574 2088 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:47.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:47.773819 2088 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.62:35093 04:09:47.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:47.776139 2088 webserver.cc:374] Webserver started at http://127.2.5.62:44763/ using document root and password file 04:09:47.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:47.777006 2088 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:09:47.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:47.780145 2150 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:09:47.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:47.781280 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 2088 04:09:47.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.2.5.61@KRBTEST.COM; defaulting to no policy 04:09:47.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:47.791744 2150 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } has no permanent_uuid. Determining permanent_uuid... 04:09:47.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:47.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.2.5.61@KRBTEST.COM" created. 04:09:47.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for kudu/127.2.5.62@KRBTEST.COM 04:09:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.2.5.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab. 04:09:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.2.5.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab. 04:09:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.2.5.61 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab. 04:09:47.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.2.5.61 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab. 04:09:47.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:47.820348 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:47.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:47.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:47.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:09:47.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:47.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:47.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:47.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:09:47.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:47.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:47.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:47.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:47.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:47.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:47.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:47.821676 2150 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } has no permanent_uuid. Determining permanent_uuid... 04:09:47.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.823572 2150 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.61:35161: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:09:47.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:47.842979 2150 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } attempt: 1 04:09:47.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.844655 2150 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.61:35161: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:09:47.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:47.892935 2150 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } attempt: 2 04:09:47.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.894647 2150 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.61:35161: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:09:47.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:47.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.969746 2155 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:47.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.970057 2155 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.61 04:09:47.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.970288 2155 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:47.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.981089 2155 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:09:47.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.981235 2155 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:47.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.981343 2155 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:47.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.981432 2155 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:47.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.981539 2155 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:47.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:47.981673 2155 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.61 04:09:48.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.000342 2155 master_main.cc:98] Master server non-default flags: 04:09:48.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:09:48.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:48.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:48.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:48.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:09:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:48.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.005923 2150 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } attempt: 3 04:09:48.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:48.007489 2150 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.61:35161: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:09:48.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.015874 2155 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 131016us 04:09:48.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:48 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608588, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:48.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.018141 2155 init.cc:359] Logged in from keytab as kudu/127.2.5.61@KRBTEST.COM (short username kudu) 04:09:48.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.020851 2155 fs_manager.cc:263] Metadata directory not provided 04:09:48.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.020992 2155 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:48.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.021302 2155 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:09:48.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.021421 2155 server_base.cc:449] Attempting to create new FS layout instead 04:09:48.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.024370 2155 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/instance: 04:09:48.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "513938f6acf044ed88d2d88de9565cad" 04:09:48.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:48 on dist-test-slave-dist-test-slave-fvsv" 04:09:48.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.025166 2155 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal/instance: 04:09:48.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "513938f6acf044ed88d2d88de9565cad" 04:09:48.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:48 on dist-test-slave-dist-test-slave-fvsv" 04:09:48.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.033979 2155 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s 04:09:48.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.035243 2155 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:48.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.035780 2155 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.042362 2155 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:09:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.042495 2155 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "513938f6acf044ed88d2d88de9565cad" 04:09:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:48 on dist-test-slave-dist-test-slave-fvsv" 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.042865 2155 fs_report.cc:352] FS layout report 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/data 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:48.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.183787 2150 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } attempt: 4 04:09:48.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:48.185418 2150 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.61:35161: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:09:48.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.488705 2150 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } attempt: 5 04:09:48.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:48.490353 2150 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.61:35161: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:09:48.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:48.911437 2155 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.845s user 0.343s sys 0.501s 04:09:48.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:48.911650 2155 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.845s user 0.343s sys 0.501s 04:09:48.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:48.965716 2155 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:49.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.024637 2150 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } attempt: 6 04:09:49.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.028419 2150 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.61:35161: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:09:49.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.036388 2155 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.61:35161 04:09:49.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.039183 2155 webserver.cc:374] Webserver started at http://127.2.5.61:45345/ using document root and password file 04:09:49.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.040035 2155 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:09:49.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.043640 2218 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:09:49.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.047150 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 2155 04:09:49.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.055368 2218 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } has no permanent_uuid. Determining permanent_uuid... 04:09:49.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.2.5.60@KRBTEST.COM; defaulting to no policy 04:09:49.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:49.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.2.5.60@KRBTEST.COM" created. 04:09:49.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:49 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608588, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for kudu/127.2.5.62@KRBTEST.COM 04:09:49.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.084668 2218 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } has no permanent_uuid. Determining permanent_uuid... 04:09:49.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:49 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608588, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for kudu/127.2.5.61@KRBTEST.COM 04:09:49.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:09:49.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.2.5.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab. 04:09:49.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.2.5.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab. 04:09:49.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.2.5.60 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab. 04:09:49.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.2.5.60 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab. 04:09:49.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.098366 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:49.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:49.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:49.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:09:49.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:49.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:49.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:49.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:09:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:49.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:49.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:49.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:49.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:49.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:49.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.104346 2218 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } has no permanent_uuid. Determining permanent_uuid... 04:09:49.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.105777 2218 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.60:39705: Network error: Client connection negotiation failed: client connection to 127.2.5.60:39705: connect: Connection refused (error 111) 04:09:49.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.125087 2218 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } attempt: 1 04:09:49.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.127192 2218 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.60:39705: Network error: Client connection negotiation failed: client connection to 127.2.5.60:39705: connect: Connection refused (error 111) 04:09:49.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.175611 2218 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } attempt: 2 04:09:49.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.177223 2218 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.60:39705: Network error: Client connection negotiation failed: client connection to 127.2.5.60:39705: connect: Connection refused (error 111) 04:09:49.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:49.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.245698 2224 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:49.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.246029 2224 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.60 04:09:49.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.246253 2224 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:49.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.257539 2224 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:09:49.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.257678 2224 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:49.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.257786 2224 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:49.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.257874 2224 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:49.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.257989 2224 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:49.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.258111 2224 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.60 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.276804 2224 master_main.cc:98] Master server non-default flags: 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:49.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:49.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:49.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:09:49.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:49.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:49.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:49.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:49.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:49.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:49.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:49.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:49.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:49.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.288583 2218 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } attempt: 3 04:09:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.290217 2218 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.60:39705: Network error: Client connection negotiation failed: client connection to 127.2.5.60:39705: connect: Connection refused (error 111) 04:09:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.291342 2224 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 131516us 04:09:49.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:49 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608589, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:49.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.293499 2224 init.cc:359] Logged in from keytab as kudu/127.2.5.60@KRBTEST.COM (short username kudu) 04:09:49.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.296394 2224 fs_manager.cc:263] Metadata directory not provided 04:09:49.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.296545 2224 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:49.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.296856 2224 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:09:49.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.296977 2224 server_base.cc:449] Attempting to create new FS layout instead 04:09:49.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.299966 2224 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/instance: 04:09:49.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:09:49.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:49 on dist-test-slave-dist-test-slave-fvsv" 04:09:49.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.300782 2224 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal/instance: 04:09:49.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:09:49.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:49 on dist-test-slave-dist-test-slave-fvsv" 04:09:49.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.310082 2224 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s 04:09:49.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.311359 2224 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:49.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.311923 2224 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:49.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.318552 2224 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:09:49.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.318678 2224 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:49 on dist-test-slave-dist-test-slave-fvsv" 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.318933 2224 fs_report.cc:352] FS layout report 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/data 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:49.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:49.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:49.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:49.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.466565 2218 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } attempt: 4 04:09:49.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.468214 2218 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.60:39705: Network error: Client connection negotiation failed: client connection to 127.2.5.60:39705: connect: Connection refused (error 111) 04:09:49.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:49.771502 2218 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } attempt: 5 04:09:49.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:49.773133 2218 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.60:39705: Network error: Client connection negotiation failed: client connection to 127.2.5.60:39705: connect: Connection refused (error 111) 04:09:50.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.068727 2150 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } attempt: 7 04:09:50.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for kudu/127.2.5.61@KRBTEST.COM 04:09:50.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.094630 2150 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } has no permanent_uuid. Determining permanent_uuid... 04:09:50.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:50.096603 2150 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.60:39705: Network error: Client connection negotiation failed: client connection to 127.2.5.60:39705: connect: Connection refused (error 111) 04:09:50.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.148950 2150 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } attempt: 1 04:09:50.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:50.150959 2150 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.60:39705: Network error: Client connection negotiation failed: client connection to 127.2.5.60:39705: connect: Connection refused (error 111) 04:09:50.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:50.159698 2224 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.817s user 0.325s sys 0.468s 04:09:50.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:50.159967 2224 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.817s user 0.325s sys 0.468s 04:09:50.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.195343 2150 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } attempt: 2 04:09:50.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:50.197265 2150 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.2.5.60:39705: Network error: Client connection negotiation failed: client connection to 127.2.5.60:39705: connect: Connection refused (error 111) 04:09:50.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.199759 2224 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:50.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.270599 2150 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } attempt: 3 04:09:50.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.270861 2224 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.60:39705 04:09:50.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.276645 2224 webserver.cc:374] Webserver started at http://127.2.5.60:34779/ using document root and password file 04:09:50.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.277540 2224 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:09:50.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.281963 2289 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:09:50.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.285313 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 2224 04:09:50.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608586, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:09:50.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.298427 2289 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } has no permanent_uuid. Determining permanent_uuid... 04:09:50.307 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.307443 2218 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } attempt: 6 04:09:50.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608589, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for kudu/127.2.5.62@KRBTEST.COM 04:09:50.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608588, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:09:50.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:09:50.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.330559 2150 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap starting. 04:09:50.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.334070 2150 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: No blocks or log segments found. Creating new log. 04:09:50.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.336117 2150 log.cc:525] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Log is configured to *not* fsync() on all Append() calls 04:09:50.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.336818 2289 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } has no permanent_uuid. Determining permanent_uuid... 04:09:50.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.339571 2150 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: No bootstrap required, opened a new log 04:09:50.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.346381 2218 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap starting. 04:09:50.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608589, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for kudu/127.2.5.61@KRBTEST.COM 04:09:50.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.356123 2218 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: No blocks or log segments found. Creating new log. 04:09:50.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.358623 2218 log.cc:525] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Log is configured to *not* fsync() on all Append() calls 04:09:50.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.358193 2150 raft_consensus.cc:340] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:50.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.358912 2150 raft_consensus.cc:366] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:09:50.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.359150 2150 raft_consensus.cc:705] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccf046f8f4748bc9dea8828a7a48863, State: Initialized, Role: FOLLOWER 04:09:50.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.359954 2150 consensus_queue.cc:229] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:50.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.362658 2289 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } has no permanent_uuid. Determining permanent_uuid... 04:09:50.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.363991 2150 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.364595 2150 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:09:50.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.365700 2150 sys_catalog.cc:427] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: configured and running, proceeding with master startup. 04:09:50.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.368221 2218 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: No bootstrap required, opened a new log 04:09:50.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608589, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:09:50.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.61@KRBTEST.COM 04:09:50.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.391692 2289 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap starting. 04:09:50.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.400841 2289 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: No blocks or log segments found. Creating new log. 04:09:50.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.402951 2289 log.cc:525] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Log is configured to *not* fsync() on all Append() calls 04:09:50.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.422981 2289 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: No bootstrap required, opened a new log 04:09:50.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.62@KRBTEST.COM 04:09:50.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.435474 2218 raft_consensus.cc:340] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:50.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.436336 2218 raft_consensus.cc:366] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:09:50.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.436573 2218 raft_consensus.cc:705] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513938f6acf044ed88d2d88de9565cad, State: Initialized, Role: FOLLOWER 04:09:50.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.439379 2218 consensus_queue.cc:229] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:50.441 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:50.442 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_kdc_env_vars { } 04:09:50.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.441588 2068 external_mini_cluster.cc:628] 0 TS(s) registered with all masters 04:09:50.443 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: get_kdc_env_vars { env_vars { key: "KRB5_KDC_PROFILE" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf" } env_vars { key: "KRB5_CONFIG" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf" } env_vars { key: "KRB5CCNAME" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc" } env_vars { key: "KUDU_ENABLE_KRB5_REALM_FIX" value: "yes" } } 04:09:50.443 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_masters { } 04:09:50.444 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.2.5.62" port: 35093 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.2.5.61" port: 35161 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.2.5.60" port: 39705 } } } 04:09:50.445 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_tservers { } 04:09:50.445 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:50.445 [INFO - Time-limited test] (MiniKuduCluster.java:439) Running kinit for user test-admin 04:09:50.445 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: kinit { username: "test-admin" } 04:09:50.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.367650 2299 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.450950 2299 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:09:50.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.456179 2302 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.456854 2302 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:09:50.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:50.457021 2300 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:50.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:50.457252 2300 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:50.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.457303 2218 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.457845 2218 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:09:50.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.458954 2218 sys_catalog.cc:427] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: configured and running, proceeding with master startup. 04:09:50.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:50.461202 2106 tablet.cc:1799] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:09:50.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.463934 2305 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:09:50.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.464656 2289 raft_consensus.cc:340] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:50.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.465356 2289 raft_consensus.cc:366] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:09:50.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.465564 2289 raft_consensus.cc:705] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Initialized, Role: FOLLOWER 04:09:50.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.466224 2289 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:50.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.471843 2289 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.472476 2289 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:09:50.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.473736 2289 sys_catalog.cc:427] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: configured and running, proceeding with master startup. 04:09:50.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:50.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:09:50.484 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:50.535 [DEBUG - Time-limited test] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc 04:09:50.539 [DEBUG - Time-limited test] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:09:50.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:50.472712 2173 tablet.cc:1799] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:09:50.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.472077 2307 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.574177 2307 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:09:50.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:50.580031 2308 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:09:50.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:50.580282 2308 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:09:50.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.630354 2307 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:09:50.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.630828 2307 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:50.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.632936 2307 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:09:50.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.634209 2127 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" is_pre_election: true 04:09:50.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.635082 2127 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 0. 04:09:50.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.634889 2194 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513938f6acf044ed88d2d88de9565cad" is_pre_election: true 04:09:50.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.635648 2194 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 0. 04:09:50.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.636387 2239 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 04:09:50.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.637631 2307 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 0 FOLLOWER]: Leader pre-election won for term 1 04:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.637873 2307 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:09:50.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.638104 2307 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 0 FOLLOWER]: Advancing to term 1 04:09:50.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.643292 2307 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:50.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.645254 2127 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:09:50.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.645403 2194 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513938f6acf044ed88d2d88de9565cad" 04:09:50.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.645717 2127 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 0 FOLLOWER]: Advancing to term 1 04:09:50.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.645876 2194 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 0 FOLLOWER]: Advancing to term 1 04:09:50.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.650743 2127 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 1. 04:09:50.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.650836 2194 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 1. 04:09:50.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.651808 2234 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 513938f6acf044ed88d2d88de9565cad, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 04:09:50.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.653419 2307 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 1 election: Requested vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:09:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.653957 2307 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 1 FOLLOWER]: Leader election won for term 1 04:09:50.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.654439 2307 raft_consensus.cc:667] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 1 LEADER]: Becoming Leader. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Running, Role: LEADER 04:09:50.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.655164 2307 consensus_queue.cc:206] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:50.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.661244 2319 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 1 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.661999 2319 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:09:50.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.667963 2321 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:09:50.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.670152 2321 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:09:50.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.686883 2127 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 1 FOLLOWER]: Refusing update from remote peer 8a3f32f20ddc40d986bae579a55da0f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:09:50.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.686851 2194 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 1 FOLLOWER]: Refusing update from remote peer 8a3f32f20ddc40d986bae579a55da0f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:09:50.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.688196 2307 consensus_queue.cc:984] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:09:50.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.688971 2319 consensus_queue.cc:984] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:09:50.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:50.705549 2244 tablet.cc:1799] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:09:50.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.712174 2302 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 1 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.712851 2302 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:09:50.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.714769 2322 mvcc.cc:203] Tried to move safe_time back from 6396348787441377280 to 6396348787331764224. Current Snapshot: MvccSnapshot[committed={T|T < 6396348787441377280}] 04:09:50.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.718080 2321 catalog_manager.cc:959] Generated new certificate authority record 04:09:50.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.719364 2324 mvcc.cc:203] Tried to move safe_time back from 6396348787441377280 to 6396348787331764224. Current Snapshot: MvccSnapshot[committed={T|T < 6396348787441377280}] 04:09:50.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.721623 2321 catalog_manager.cc:1084] Loading token signing keys... 04:09:50.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.721279 2302 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.721913 2302 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:09:50.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.721714 2299 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 1 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.722383 2299 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:09:50.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.723300 2319 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.726362 2319 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:09:50.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.737486 2319 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.738137 2319 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:09:50.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.742286 2323 mvcc.cc:203] Tried to move safe_time back from 6396348787441377280 to 6396348787331764224. Current Snapshot: MvccSnapshot[committed={T|T < 6396348787441377280}] 04:09:50.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.748309 2299 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:50.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.748958 2299 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:09:50.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:50.755715 2321 catalog_manager.cc:4350] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Generated new TSK 0 04:09:51.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.461825 2300 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: acquiring CA information for follower catalog manager: success 04:09:51.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.464205 2300 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:09:51.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.472462 2305 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: acquiring CA information for follower catalog manager: success 04:09:51.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.475236 2305 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:09:51.540 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.60:39705 04:09:51.541 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:09:51.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.541962 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 2224 04:09:51.575 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:51.576 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.62:35093 04:09:51.576 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:09:51.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.578114 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 2088 04:09:51.608 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:51.608 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.61:35161 04:09:51.608 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:09:51.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.609158 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 2155 04:09:51.627 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:51.627 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.60:39705 04:09:51.628 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:09:51.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.629237 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:51.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:51.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:51.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:09:51.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:51.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:51.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:51.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:51.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:51.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:51.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:51.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:51.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:51.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:51.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:51.775921 2334 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:51.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:51.776247 2334 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.60 04:09:51.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:51.776474 2334 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:51.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:51.787371 2334 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:09:51.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:51.787506 2334 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:51.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:51.787611 2334 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:51.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:51.787703 2334 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:51.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:51.787820 2334 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:51.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:51.787941 2334 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.60 04:09:51.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.806322 2334 master_main.cc:98] Master server non-default flags: 04:09:51.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:09:51.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:51.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:51.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:09:51.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:51.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:51.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.821467 2334 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 132516us 04:09:51.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:51 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608591, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:51.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.823853 2334 init.cc:359] Logged in from keytab as kudu/127.2.5.60@KRBTEST.COM (short username kudu) 04:09:51.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.827069 2334 fs_manager.cc:263] Metadata directory not provided 04:09:51.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.827200 2334 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.832159 2334 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s 04:09:51.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.832775 2334 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:51.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.833256 2334 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:51.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.839910 2334 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.840029 2334 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:49 on dist-test-slave-dist-test-slave-fvsv" 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:51.841266 2334 fs_report.cc:352] FS layout report 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/data 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:51.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:51.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:52.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:52.490429 2334 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.626s user 0.260s sys 0.337s 04:09:52.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:52.490662 2334 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.626s user 0.260s sys 0.337s 04:09:52.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.552974 2334 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:52.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.618741 2334 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.60:39705 04:09:52.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.621697 2334 webserver.cc:374] Webserver started at http://127.2.5.60:34779/ using document root and password file 04:09:52.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.622625 2334 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:09:52.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.630131 2396 sys_catalog.cc:188] Verifying existing consensus state 04:09:52.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.631973 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 2334 04:09:52.632 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:52.632 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.62:35093 04:09:52.632 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:09:52.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.633548 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:52.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:52.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:52.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:09:52.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:52.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:52.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:52.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:09:52.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:09:52.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:52.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:52.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:52.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:52.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:52.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:52.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.633895 2396 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap starting. 04:09:52.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.686069 2396 log.cc:525] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Log is configured to *not* fsync() on all Append() calls 04:09:52.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.699477 2396 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:09:52.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.700346 2396 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap complete. 04:09:52.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.728524 2396 raft_consensus.cc:340] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:52.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.733721 2396 raft_consensus.cc:705] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Initialized, Role: FOLLOWER 04:09:52.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.734459 2396 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.736651 2399 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.737161 2399 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:09:52.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.737572 2396 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:52.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.737983 2396 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:09:52.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.738826 2396 sys_catalog.cc:427] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: configured and running, proceeding with master startup. 04:09:52.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.740288 2400 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:09:52.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:52.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:52.814023 2397 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:52.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:52.814370 2397 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.62 04:09:52.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:52.814697 2397 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:52.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:52.826310 2397 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:09:52.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:52.826467 2397 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:52.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:52.826607 2397 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:52.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:52.826743 2397 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:52.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:52.826917 2397 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:52.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:52.827121 2397 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.62 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.846032 2397 master_main.cc:98] Master server non-default flags: 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:52.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:52.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:52.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.860648 2397 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 133016us 04:09:52.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:52 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608592, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:52.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.863011 2397 init.cc:359] Logged in from keytab as kudu/127.2.5.62@KRBTEST.COM (short username kudu) 04:09:52.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.866927 2397 fs_manager.cc:263] Metadata directory not provided 04:09:52.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.867106 2397 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:52.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.872614 2397 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s 04:09:52.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.873296 2397 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:52.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.873831 2397 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:52.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.880941 2397 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s 04:09:52.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.881078 2397 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:52.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:09:52.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:46 on dist-test-slave-dist-test-slave-fvsv" 04:09:52.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:52.882423 2397 fs_report.cc:352] FS layout report 04:09:52.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:52.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:52.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:52.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/data 04:09:52.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:52.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:52.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:52.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:52.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:52.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:52.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:52.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:52.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:52.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:53.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:53.510923 2397 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.604s user 0.223s sys 0.380s 04:09:53.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:53.511217 2397 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.604s user 0.223s sys 0.380s 04:09:53.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.573222 2397 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:53.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.642012 2397 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.62:35093 04:09:53.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.644852 2397 webserver.cc:374] Webserver started at http://127.2.5.62:44763/ using document root and password file 04:09:53.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.645799 2397 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:09:53.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.653297 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 2397 04:09:53.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.653344 2462 sys_catalog.cc:188] Verifying existing consensus state 04:09:53.653 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:53.653 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.61:35161 04:09:53.654 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:09:53.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.655441 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:53.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:53.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:53.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:09:53.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:53.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:09:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:09:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:53.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:53.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.658208 2462 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap starting. 04:09:53.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.702075 2462 log.cc:525] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Log is configured to *not* fsync() on all Append() calls 04:09:53.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.729097 2462 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:09:53.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.729719 2462 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap complete. 04:09:53.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.749311 2400 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: acquiring CA information for follower catalog manager: success 04:09:53.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.752225 2400 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:09:53.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.753998 2462 raft_consensus.cc:340] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:53.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.759299 2462 raft_consensus.cc:705] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccf046f8f4748bc9dea8828a7a48863, State: Initialized, Role: FOLLOWER 04:09:53.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.760033 2462 consensus_queue.cc:229] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:53.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.763819 2462 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:53.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.764530 2462 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:09:53.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.765620 2462 sys_catalog.cc:427] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: configured and running, proceeding with master startup. 04:09:53.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.764210 2466 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:53.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.871407 2466 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:09:53.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.885231 2467 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: acquiring CA information for follower catalog manager: success 04:09:53.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:53.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:53.886924 2463 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:53.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:53.887650 2463 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.61 04:09:53.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:53.888170 2463 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:53.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.888581 2467 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:09:53.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:53.907778 2463 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:09:53.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:53.907999 2463 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:53.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:53.908180 2463 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:53.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:53.908330 2463 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:53.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:53.908524 2463 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:53.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:53.908721 2463 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.61 04:09:53.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.951840 2463 master_main.cc:98] Master server non-default flags: 04:09:53.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:09:53.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:53.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:53.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:53.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:09:53.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:53.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:53.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:09:53.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:53.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:53.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:09:53.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:53.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:53.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:53.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:53.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:53.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:53.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:53.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:53.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:53.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.975920 2463 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 133516us 04:09:53.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:53 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608593, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:53.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.979182 2463 init.cc:359] Logged in from keytab as kudu/127.2.5.61@KRBTEST.COM (short username kudu) 04:09:53.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.983492 2463 fs_manager.cc:263] Metadata directory not provided 04:09:53.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.983695 2463 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:53.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.990841 2463 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s 04:09:53.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.991755 2463 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:53.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:53.992463 2463 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:54.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.012681 2463 fs_manager.cc:419] Time spent opening block manager: real 0.012s user 0.008s sys 0.003s 04:09:54.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.012876 2463 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:54.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "513938f6acf044ed88d2d88de9565cad" 04:09:54.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:48 on dist-test-slave-dist-test-slave-fvsv" 04:09:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.014791 2463 fs_report.cc:352] FS layout report 04:09:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/data 04:09:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:54.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:54.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:54.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.488258 2481 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:09:54.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.626282 2481 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:54.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.631734 2481 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:09:54.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:54.644989 2344 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:09:54.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.661099 2439 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" is_pre_election: true 04:09:54.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.661916 2439 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 1. 04:09:54.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.663467 2347 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:09:54.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.664567 2481 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 1 FOLLOWER]: Leader pre-election won for term 2 04:09:54.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.664811 2481 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:09:54.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.665036 2481 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 1 FOLLOWER]: Advancing to term 2 04:09:54.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.672199 2481 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:54.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.674319 2439 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:09:54.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.674800 2439 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 1 FOLLOWER]: Advancing to term 2 04:09:54.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:54.682174 2344 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:09:54.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.682577 2481 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 2 election: Requested vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:09:54.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.683677 2439 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 2. 04:09:54.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.684556 2347 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:09:54.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.685401 2481 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 2 FOLLOWER]: Leader election won for term 2 04:09:54.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.687690 2481 raft_consensus.cc:667] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 2 LEADER]: Becoming Leader. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Running, Role: LEADER 04:09:54.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.688241 2481 consensus_queue.cc:206] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:54.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.700028 2485 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 2 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:54.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.700785 2485 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:09:54.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.703228 2487 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:09:54.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.707254 2487 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:09:54.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.709972 2487 catalog_manager.cc:1084] Loading token signing keys... 04:09:54.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:54.711865 2487 catalog_manager.cc:4361] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Loaded TSK: 0 04:09:55.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.155166 2439 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 2 FOLLOWER]: Refusing update from remote peer 8a3f32f20ddc40d986bae579a55da0f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) 04:09:55.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.156657 2485 consensus_queue.cc:984] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s 04:09:55.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.163944 2488 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 2 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:55.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.164691 2488 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:09:55.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:55.179788 2344 consensus_peers.cc:458] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 -> Peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Couldn't send request to peer 513938f6acf044ed88d2d88de9565cad. Status: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:09:55.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.181176 2485 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:55.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.182107 2485 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:09:55.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.185927 2488 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:55.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.186795 2488 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:09:55.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:55.297523 2463 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 1.238s user 0.390s sys 0.277s 04:09:55.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:55.297789 2463 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 1.238s user 0.390s sys 0.277s 04:09:55.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.368429 2463 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:55.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.497056 2463 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.61:35161 04:09:55.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.499509 2463 webserver.cc:374] Webserver started at http://127.2.5.61:45345/ using document root and password file 04:09:55.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.500447 2463 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:09:55.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.502985 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 2463 04:09:55.503 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:55.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.509563 2541 sys_catalog.cc:188] Verifying existing consensus state 04:09:55.520 [DEBUG - New I/O worker #55] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:09:55.521 [DEBUG - New I/O worker #55] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc 04:09:55.522 [DEBUG - New I/O worker #55] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:09:55.523 [INFO - New I/O worker #55] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:09:55.524 [DEBUG - New I/O worker #56] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:09:55.524 [DEBUG - New I/O worker #56] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc 04:09:55.525 [DEBUG - New I/O worker #56] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:09:55.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.525960 2541 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap starting. 04:09:55.526 [INFO - New I/O worker #56] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:09:55.527 [DEBUG - New I/O worker #56] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.2.5.62 04:09:55.533 [DEBUG - New I/O worker #57] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:09:55.534 [DEBUG - New I/O worker #57] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc 04:09:55.535 [DEBUG - New I/O worker #57] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:09:55.535 [INFO - New I/O worker #57] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:09:55.536 [DEBUG - New I/O worker #57] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.2.5.61 04:09:55.547 [DEBUG - New I/O worker #55] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.2.5.60 04:09:55.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.558164 2541 log.cc:525] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Log is configured to *not* fsync() on all Append() calls 04:09:55.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.575803 2541 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:09:55.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.576650 2541 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap complete. 04:09:55.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:55 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.62@KRBTEST.COM 04:09:55.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:55 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:09:55.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:55 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.61@KRBTEST.COM 04:09:55.607 [DEBUG - New I/O worker #56] (Negotiator.java:764) Authenticated connection [id: 0x370b20c8, /127.0.0.1:57902 => /127.2.5.62:35093] using SASL/GSSAPI 04:09:55.619 [DEBUG - New I/O worker #57] (Negotiator.java:764) Authenticated connection [id: 0xb7cb3d48, /127.0.0.1:35972 => /127.2.5.61:35161] using SASL/GSSAPI 04:09:55.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.599098 2541 raft_consensus.cc:340] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:55.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.622350 2541 raft_consensus.cc:705] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513938f6acf044ed88d2d88de9565cad, State: Initialized, Role: FOLLOWER 04:09:55.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.623401 2541 consensus_queue.cc:229] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:55.624 [INFO - New I/O worker #57] (ConnectToCluster.java:361) Unable to connect to master 127.2.5.61:35161: Catalog manager is not initialized. State: Starting 04:09:55.625 [DEBUG - New I/O worker #55] (Negotiator.java:764) Authenticated connection [id: 0xf7f8355c, /127.0.0.1:58390 => /127.2.5.60:39705] using SASL/GSSAPI 04:09:55.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.627954 2541 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:55.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.630144 2541 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:09:55.632 [DEBUG - New I/O worker #55] (AsyncKuduClient.java:2322) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 04:09:55.633 [DEBUG - New I/O worker #55] (AsyncKuduClient.java:1296) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=129), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.5.60:39705, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.5.62:35093, [4ms] Sub RPC ConnectToMaster: sending RPC to server master-127.2.5.61:35161, [110ms] Sub RPC ConnectToMaster: received response from server master-127.2.5.62:35093: OK, [117ms] Sub RPC ConnectToMaster: received response from server master-127.2.5.61:35161: Service unavailable: Catalog manager is not initialized. State: Starting (error 0), [124ms] Sub RPC ConnectToMaster: received response from server master-127.2.5.60:39705: OK, deferred=null) after lookup 04:09:55.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.635690 2541 sys_catalog.cc:427] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: configured and running, proceeding with master startup. 04:09:55.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:55 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608591, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for kudu/127.2.5.61@KRBTEST.COM 04:09:55.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.629396 2547 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:55.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.708021 2547 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:09:55.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.712898 2518 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 1 FOLLOWER]: Advancing to term 2 04:09:55.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.720757 2548 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: acquiring CA information for follower catalog manager: success 04:09:55.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.723976 2548 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:09:55.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.728996 2547 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 2 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:55.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.729697 2547 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:09:55.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.730479 2553 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:55.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:55.731225 2553 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:09:56.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.218797 2481 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:56.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.219553 2481 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:09:56.635 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.60:39705 04:09:56.635 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:09:56.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.637960 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 2334 04:09:56.662 [DEBUG - New I/O worker #55] (Connection.java:691) [peer master-127.2.5.60:39705(127.2.5.60:39705)] cleaning up while in state READY due to: connection disconnected 04:09:56.663 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:56.664 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.62:35093 04:09:56.664 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:09:56.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.665216 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 2397 04:09:56.691 [DEBUG - New I/O worker #56] (Connection.java:691) [peer master-127.2.5.62:35093(127.2.5.62:35093)] cleaning up while in state READY due to: connection disconnected 04:09:56.691 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:56.692 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.61:35161 04:09:56.692 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:09:56.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.692917 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 2463 04:09:56.709 [DEBUG - New I/O worker #57] (Connection.java:691) [peer master-127.2.5.61:35161(127.2.5.61:35161)] cleaning up while in state READY due to: connection disconnected 04:09:56.710 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:56.710 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.60:39705 04:09:56.710 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.712925 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:56.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:56.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:56.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:56.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:56.862576 2558 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:56.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:56.862918 2558 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.60 04:09:56.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:56.863176 2558 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:56.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:56.874029 2558 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:09:56.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:56.874166 2558 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:56.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:56.874292 2558 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:56.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:56.874418 2558 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:56.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:56.874593 2558 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:56.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:56.874770 2558 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.60 04:09:56.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.893635 2558 master_main.cc:98] Master server non-default flags: 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:56.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:56.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:56.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:56.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:56.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:56.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:56.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:56.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:56.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.908231 2558 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 135016us 04:09:56.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:56 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:56.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.910439 2558 init.cc:359] Logged in from keytab as kudu/127.2.5.60@KRBTEST.COM (short username kudu) 04:09:56.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.914090 2558 fs_manager.cc:263] Metadata directory not provided 04:09:56.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.914211 2558 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:56.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.919395 2558 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s 04:09:56.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.920047 2558 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:56.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.920537 2558 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:56.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.927201 2558 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:09:56.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.927322 2558 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:56.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:09:56.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:49 on dist-test-slave-dist-test-slave-fvsv" 04:09:56.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:56.928613 2558 fs_report.cc:352] FS layout report 04:09:56.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:56.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:56.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:09:56.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/data 04:09:56.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:56.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:56.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:56.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:56.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:56.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:56.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:56.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:56.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:56.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:57.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:57.565275 2558 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.606s user 0.001s sys 0.001s 04:09:57.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:57.565475 2558 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.606s user 0.001s sys 0.001s 04:09:57.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.589723 2558 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:57.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.665357 2558 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.60:39705 04:09:57.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.668592 2558 webserver.cc:374] Webserver started at http://127.2.5.60:34779/ using document root and password file 04:09:57.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.669826 2558 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:09:57.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.676956 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 2558 04:09:57.677 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:57.677 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.62:35093 04:09:57.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.677757 2620 sys_catalog.cc:188] Verifying existing consensus state 04:09:57.677 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.679162 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:57.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:09:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:09:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:57.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.683460 2620 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap starting. 04:09:57.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.742913 2620 log.cc:525] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Log is configured to *not* fsync() on all Append() calls 04:09:57.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.762571 2620 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:09:57.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.763293 2620 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap complete. 04:09:57.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.783073 2620 raft_consensus.cc:340] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:57.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.788183 2620 raft_consensus.cc:705] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Initialized, Role: FOLLOWER 04:09:57.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.788920 2620 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:57.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.790930 2620 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:57.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.791417 2620 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:09:57.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.792474 2620 sys_catalog.cc:427] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: configured and running, proceeding with master startup. 04:09:57.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.791402 2623 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:57.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.861121 2623 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:09:57.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.873317 2624 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: acquiring CA information for follower catalog manager: success 04:09:57.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.876539 2624 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:09:57.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:57.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:57.902199 2621 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:57.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:57.902565 2621 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.62 04:09:57.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:57.902895 2621 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:57.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:57.913986 2621 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:09:57.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:57.914142 2621 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:57.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:57.914291 2621 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:57.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:57.914427 2621 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:57.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:57.914613 2621 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:57.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:57.914788 2621 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.62 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.934875 2621 master_main.cc:98] Master server non-default flags: 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:09:57.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:57.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:57.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:57.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:57.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:57.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:57.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:57.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.949604 2621 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 135516us 04:09:57.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:57 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608597, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:57.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.952071 2621 init.cc:359] Logged in from keytab as kudu/127.2.5.62@KRBTEST.COM (short username kudu) 04:09:57.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.955579 2621 fs_manager.cc:263] Metadata directory not provided 04:09:57.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.955726 2621 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:57.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.960618 2621 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s 04:09:57.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.961308 2621 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:57.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.961843 2621 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:57.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.968768 2621 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s 04:09:57.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.968914 2621 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:57.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:09:57.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:46 on dist-test-slave-dist-test-slave-fvsv" 04:09:57.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:57.970254 2621 fs_report.cc:352] FS layout report 04:09:57.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:57.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:57.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:09:57.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/data 04:09:57.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:57.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:57.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:57.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:57.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:57.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:57.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:57.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:57.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:57.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:09:58.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:58.872115 2621 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.876s user 0.315s sys 0.518s 04:09:58.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:58.872344 2621 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.877s user 0.315s sys 0.518s 04:09:58.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:58.952777 2621 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:09:58.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:58.980391 2646 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:09:58.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:58.980861 2646 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:59.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:59.000488 2568 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:09:59.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.001188 2646 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:09:59.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:59.001824 2571 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093): Network error: Client connection negotiation failed: client connection to 127.2.5.62:35093: connect: Connection refused (error 111) 04:09:59.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.002152 2571 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad, 7ccf046f8f4748bc9dea8828a7a48863 04:09:59.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.003180 2646 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority 04:09:59.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.041600 2621 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.62:35093 04:09:59.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.044088 2621 webserver.cc:374] Webserver started at http://127.2.5.62:44763/ using document root and password file 04:09:59.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.045078 2621 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:09:59.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.049541 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 2621 04:09:59.050 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:09:59.050 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.61:35161 04:09:59.050 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:09:59.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.052561 2690 sys_catalog.cc:188] Verifying existing consensus state 04:09:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.053539 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:09:59.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:59.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:59.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:09:59.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.057250 2690 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap starting. 04:09:59.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.084698 2690 log.cc:525] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Log is configured to *not* fsync() on all Append() calls 04:09:59.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.093523 2690 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:09:59.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.094202 2690 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap complete. 04:09:59.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.113816 2690 raft_consensus.cc:340] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:59.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.118965 2690 raft_consensus.cc:705] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccf046f8f4748bc9dea8828a7a48863, State: Initialized, Role: FOLLOWER 04:09:59.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.119702 2690 consensus_queue.cc:229] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:09:59.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.121794 2693 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:59.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.122305 2693 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:09:59.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.122699 2690 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:09:59.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.123142 2690 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:09:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.124060 2690 sys_catalog.cc:427] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: configured and running, proceeding with master startup. 04:09:59.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.140447 2694 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: acquiring CA information for follower catalog manager: success 04:09:59.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.147430 2694 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:09:59.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:09:59.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:59.245231 2691 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:09:59.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:59.245719 2691 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.61 04:09:59.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:59.246080 2691 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:09:59.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:59.265393 2691 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:09:59.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:59.265600 2691 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:09:59.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:59.265782 2691 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:09:59.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:59.265944 2691 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:09:59.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:59.266129 2691 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:09:59.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:09:59.266324 2691 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.61 04:09:59.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.299587 2691 master_main.cc:98] Master server non-default flags: 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:09:59.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:09:59.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:09:59.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:09:59.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:09:59.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:09:59.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.313652 2691 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 136516us 04:09:59.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:09:59 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:09:59.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.316118 2691 init.cc:359] Logged in from keytab as kudu/127.2.5.61@KRBTEST.COM (short username kudu) 04:09:59.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.323787 2691 fs_manager.cc:263] Metadata directory not provided 04:09:59.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.323935 2691 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:09:59.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.332453 2691 fs_manager.cc:399] Time spent opening directory manager: real 0.006s user 0.004s sys 0.004s 04:09:59.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.333170 2691 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:09:59.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.333737 2691 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:09:59.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.344333 2691 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s 04:09:59.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.344470 2691 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:59.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "513938f6acf044ed88d2d88de9565cad" 04:09:59.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:48 on dist-test-slave-dist-test-slave-fvsv" 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:09:59.345818 2691 fs_report.cc:352] FS layout report 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/data 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:09:59.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:00.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:00.130333 2691 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.757s user 0.341s sys 0.404s 04:10:00.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:00.130614 2691 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.758s user 0.341s sys 0.404s 04:10:00.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.184528 2691 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:00.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.304411 2740 raft_consensus.cc:472] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:10:00.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.304864 2740 raft_consensus.cc:494] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:00.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.307999 2740 leader_election.cc:296] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161), 8a3f32f20ddc40d986bae579a55da0f1 (127.2.5.60:39705) 04:10:00.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:00.312603 2635 leader_election.cc:341] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:00.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.329861 2597 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ccf046f8f4748bc9dea8828a7a48863" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8a3f32f20ddc40d986bae579a55da0f1" is_pre_election: true 04:10:00.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.330500 2597 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ccf046f8f4748bc9dea8828a7a48863 in term 2. 04:10:00.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.331673 2635 leader_election.cc:310] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:10:00.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.332376 2740 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 2 FOLLOWER]: Leader pre-election won for term 3 04:10:00.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.332463 2691 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.61:35161 04:10:00.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.332617 2740 raft_consensus.cc:472] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:10:00.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.332770 2740 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 2 FOLLOWER]: Advancing to term 3 04:10:00.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.335855 2691 webserver.cc:374] Webserver started at http://127.2.5.61:45345/ using document root and password file 04:10:00.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.336982 2691 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:00.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.337916 2740 raft_consensus.cc:494] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:00.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.339396 2740 leader_election.cc:296] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [CANDIDATE]: Term 3 election: Requested vote from peers 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161), 8a3f32f20ddc40d986bae579a55da0f1 (127.2.5.60:39705) 04:10:00.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.340628 2597 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ccf046f8f4748bc9dea8828a7a48863" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:10:00.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.341121 2597 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 2 FOLLOWER]: Advancing to term 3 04:10:00.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.345820 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 2691 04:10:00.346 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:00.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.348489 2597 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ccf046f8f4748bc9dea8828a7a48863 in term 3. 04:10:00.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.349671 2761 sys_catalog.cc:188] Verifying existing consensus state 04:10:00.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.349658 2635 leader_election.cc:310] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 04:10:00.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.350334 2740 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 3 FOLLOWER]: Leader election won for term 3 04:10:00.350 [DEBUG - New I/O worker #58] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:10:00.352 [DEBUG - New I/O worker #58] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc 04:10:00.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.355782 2740 raft_consensus.cc:667] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 3 LEADER]: Becoming Leader. State: Replica: 7ccf046f8f4748bc9dea8828a7a48863, State: Running, Role: LEADER 04:10:00.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.356204 2761 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap starting. 04:10:00.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.356456 2740 consensus_queue.cc:206] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:00.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.364413 2763 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ccf046f8f4748bc9dea8828a7a48863. Latest consensus state: current_term: 3 leader_uuid: "7ccf046f8f4748bc9dea8828a7a48863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:00.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.364949 2763 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: LEADER 04:10:00.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.367614 2765 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:10:00.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:00 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:00.371 [DEBUG - New I/O worker #58] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:10:00.371 [INFO - New I/O worker #58] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:10:00.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.371809 2765 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:10:00.372 [DEBUG - New I/O worker #58] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.2.5.60 04:10:00.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.374567 2765 catalog_manager.cc:1084] Loading token signing keys... 04:10:00.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.376541 2765 catalog_manager.cc:4361] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Loaded TSK: 0 04:10:00.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:00 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:10:00.403 [DEBUG - New I/O worker #58] (Negotiator.java:764) Authenticated connection [id: 0x94ffd723, /127.0.0.1:58408 => /127.2.5.60:39705] using SASL/GSSAPI 04:10:00.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:00 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608597, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for kudu/127.2.5.61@KRBTEST.COM 04:10:00.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.422175 2761 log.cc:525] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Log is configured to *not* fsync() on all Append() calls 04:10:00.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.432045 2761 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:00.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.432725 2761 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap complete. 04:10:00.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.445415 2761 raft_consensus.cc:340] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:00.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.450685 2761 raft_consensus.cc:705] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513938f6acf044ed88d2d88de9565cad, State: Initialized, Role: FOLLOWER 04:10:00.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.451375 2761 consensus_queue.cc:229] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:00.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.453384 2767 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:00.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.453891 2767 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:00.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.454282 2761 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:00.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.454699 2761 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:00.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.455615 2761 sys_catalog.cc:427] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: configured and running, proceeding with master startup. 04:10:00.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.455775 2734 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7ccf046f8f4748bc9dea8828a7a48863" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "513938f6acf044ed88d2d88de9565cad" 04:10:00.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.554013 2734 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 2 FOLLOWER]: Advancing to term 3 04:10:00.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.559433 2734 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ccf046f8f4748bc9dea8828a7a48863 in term 3. 04:10:00.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.570766 2768 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: acquiring CA information for follower catalog manager: success 04:10:00.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.572726 2768 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:00.806 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.806350 2734 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 3 FOLLOWER]: Refusing update from remote peer 7ccf046f8f4748bc9dea8828a7a48863: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) 04:10:00.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.807824 2763 consensus_queue.cc:984] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s 04:10:00.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.816591 2767 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ccf046f8f4748bc9dea8828a7a48863. Latest consensus state: current_term: 3 leader_uuid: "7ccf046f8f4748bc9dea8828a7a48863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:00.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.817287 2767 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:00.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.820377 2740 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7ccf046f8f4748bc9dea8828a7a48863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:00.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.821003 2740 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: LEADER 04:10:00.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.822980 2597 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 3 FOLLOWER]: Refusing update from remote peer 7ccf046f8f4748bc9dea8828a7a48863: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) 04:10:00.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.824296 2763 consensus_queue.cc:984] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s 04:10:00.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.824928 2767 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7ccf046f8f4748bc9dea8828a7a48863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:00.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.825575 2767 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:00.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.843956 2776 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7ccf046f8f4748bc9dea8828a7a48863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:00.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.844954 2776 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:00.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.844794 2773 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ccf046f8f4748bc9dea8828a7a48863. Latest consensus state: current_term: 3 leader_uuid: "7ccf046f8f4748bc9dea8828a7a48863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:00.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:00.845561 2773 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:01.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.262884 2763 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7ccf046f8f4748bc9dea8828a7a48863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:01.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.263749 2763 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: LEADER 04:10:01.408 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.60:39705 04:10:01.408 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:10:01.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.409263 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 2558 04:10:01.428 [DEBUG - New I/O worker #58] (Connection.java:691) [peer master-127.2.5.60:39705(127.2.5.60:39705)] cleaning up while in state READY due to: connection disconnected 04:10:01.429 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:01.430 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.62:35093 04:10:01.430 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:10:01.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.430874 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 2621 04:10:01.456 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:01.456 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.61:35161 04:10:01.456 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:10:01.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.457109 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 2691 04:10:01.475 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:01.476 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.60:39705 04:10:01.476 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:10:01.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.477433 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:01.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:01.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:01.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:01.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:01.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:01.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:01.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:01.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:01.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:01.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:01.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:01.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:01.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:01.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:01.627218 2778 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:01.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:01.627660 2778 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.60 04:10:01.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:01.628055 2778 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:01.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:01.639078 2778 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:01.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:01.639219 2778 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:01.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:01.639328 2778 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:01.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:01.639420 2778 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:01.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:01.639531 2778 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:01.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:01.639653 2778 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.60 04:10:01.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.657310 2778 master_main.cc:98] Master server non-default flags: 04:10:01.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:01.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:01.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:01.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:01.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:01.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:01.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:01.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:01.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:01.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:01.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:01.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:01.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:01.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:01.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:01.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:01.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:01.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:01.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:01.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.671741 2778 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 137516us 04:10:01.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:01 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608601, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:01.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.674019 2778 init.cc:359] Logged in from keytab as kudu/127.2.5.60@KRBTEST.COM (short username kudu) 04:10:01.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.677109 2778 fs_manager.cc:263] Metadata directory not provided 04:10:01.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.677233 2778 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:01.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.682174 2778 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s 04:10:01.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.682803 2778 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:01.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.683337 2778 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:01.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.690114 2778 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:10:01.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.690235 2778 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:01.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:10:01.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:49 on dist-test-slave-dist-test-slave-fvsv" 04:10:01.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:01.691545 2778 fs_report.cc:352] FS layout report 04:10:01.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:01.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:01.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:01.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/data 04:10:01.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:01.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:01.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:01.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:01.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:01.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:01.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:01.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:01.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:01.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:02.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:02.566794 2778 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.851s user 0.270s sys 0.580s 04:10:02.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:02.567026 2778 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.851s user 0.270s sys 0.580s 04:10:02.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.619796 2778 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:02.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.687922 2778 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.60:39705 04:10:02.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.690212 2778 webserver.cc:374] Webserver started at http://127.2.5.60:34779/ using document root and password file 04:10:02.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.691154 2778 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:02.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.697098 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 2778 04:10:02.697 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:02.697 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.62:35093 04:10:02.697 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:10:02.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.698216 2840 sys_catalog.cc:188] Verifying existing consensus state 04:10:02.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.698666 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:02.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:02.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:02.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:02.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.702751 2840 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap starting. 04:10:02.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.769313 2840 log.cc:525] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Log is configured to *not* fsync() on all Append() calls 04:10:02.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.789883 2840 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:02.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.790668 2840 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap complete. 04:10:02.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.821030 2840 raft_consensus.cc:340] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:02.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.829100 2840 raft_consensus.cc:705] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Initialized, Role: FOLLOWER 04:10:02.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.829927 2840 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:02.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.832274 2840 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:02.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.832937 2840 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:02.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.834039 2840 sys_catalog.cc:427] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: configured and running, proceeding with master startup. 04:10:02.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:02.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:02.901070 2841 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:02.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:02.901439 2841 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.62 04:10:02.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:02.901768 2841 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:02.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:02.912900 2841 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:02.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:02.913044 2841 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:02.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:02.913190 2841 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:02.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:02.913328 2841 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:02.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:02.913498 2841 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:02.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:02.913682 2841 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.62 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.942834 2841 master_main.cc:98] Master server non-default flags: 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:02.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:02.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:02.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:02.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:02.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:02.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:02.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.832592 2843 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:02.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.948635 2843 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:02.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.962615 2844 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: acquiring CA information for follower catalog manager: success 04:10:02.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.964763 2844 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:02.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.965878 2841 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 138016us 04:10:02.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:02 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608602, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:02.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.969118 2841 init.cc:359] Logged in from keytab as kudu/127.2.5.62@KRBTEST.COM (short username kudu) 04:10:02.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.973467 2841 fs_manager.cc:263] Metadata directory not provided 04:10:02.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.973786 2841 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:02.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.980304 2841 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s 04:10:02.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.981003 2841 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:02.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.981571 2841 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:02.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.988376 2841 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:10:02.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.988508 2841 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:02.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:02.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:46 on dist-test-slave-dist-test-slave-fvsv" 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:02.989825 2841 fs_report.cc:352] FS layout report 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/data 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:02.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:03.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:03.680490 2841 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.657s user 0.242s sys 0.369s 04:10:03.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:03.680775 2841 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.657s user 0.242s sys 0.369s 04:10:03.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:03.744499 2841 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:03.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:03.927529 2841 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.62:35093 04:10:03.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:03.931509 2841 webserver.cc:374] Webserver started at http://127.2.5.62:44763/ using document root and password file 04:10:03.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:03.932585 2841 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:03.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:03.938716 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 2841 04:10:03.939 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:03.939 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.61:35161 04:10:03.939 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:10:03.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:03.940402 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:03.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:03.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:03.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:03.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:03.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:03.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:03.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:03.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:03.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:03.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:03.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:03.946102 2907 sys_catalog.cc:188] Verifying existing consensus state 04:10:03.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:03.958107 2907 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap starting. 04:10:04.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.045217 2907 log.cc:525] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Log is configured to *not* fsync() on all Append() calls 04:10:04.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.067629 2907 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:04.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.068368 2907 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap complete. 04:10:04.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.095429 2907 raft_consensus.cc:340] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:04.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.101141 2907 raft_consensus.cc:705] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccf046f8f4748bc9dea8828a7a48863, State: Initialized, Role: FOLLOWER 04:10:04.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.101918 2907 consensus_queue.cc:229] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:04.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.107743 2907 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:04.108 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.108305 2907 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:04.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.109443 2907 sys_catalog.cc:427] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: configured and running, proceeding with master startup. 04:10:04.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.108546 2910 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:04.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.179472 2910 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:04.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.192561 2911 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: acquiring CA information for follower catalog manager: success 04:10:04.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.199589 2911 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:04.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:04.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:04.203445 2908 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:04.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:04.203871 2908 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.61 04:10:04.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:04.204241 2908 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:04.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:04.223587 2908 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:04.223 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:04.223794 2908 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:04.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:04.223963 2908 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:04.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:04.224112 2908 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:04.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:04.224303 2908 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:04.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:04.224496 2908 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.61 04:10:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.267808 2908 master_main.cc:98] Master server non-default flags: 04:10:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:04.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:04.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:04.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:04.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:04.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:04.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:04.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:04.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:04.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:04.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.290832 2908 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 139016us 04:10:04.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:04 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:04.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.293684 2908 init.cc:359] Logged in from keytab as kudu/127.2.5.61@KRBTEST.COM (short username kudu) 04:10:04.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.305693 2908 fs_manager.cc:263] Metadata directory not provided 04:10:04.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.305886 2908 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:04.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.316952 2908 fs_manager.cc:399] Time spent opening directory manager: real 0.008s user 0.009s sys 0.000s 04:10:04.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.317885 2908 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:04.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.318596 2908 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:04.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.332605 2908 fs_manager.cc:419] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s 04:10:04.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.332803 2908 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:04.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "513938f6acf044ed88d2d88de9565cad" 04:10:04.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:48 on dist-test-slave-dist-test-slave-fvsv" 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.334659 2908 fs_report.cc:352] FS layout report 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/data 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:04.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:04.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:04.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:04.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.704093 2925 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:10:04.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.704635 2925 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:04.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:04.728657 2788 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:04.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.729560 2925 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.771713 2884 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" is_pre_election: true 04:10:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.772624 2884 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 3. 04:10:04.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.774672 2791 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:10:04.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.776218 2925 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 3 FOLLOWER]: Leader pre-election won for term 4 04:10:04.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.776500 2925 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:10:04.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.776749 2925 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 3 FOLLOWER]: Advancing to term 4 04:10:04.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.784509 2925 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:04.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.786952 2884 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:04.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.787524 2884 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 3 FOLLOWER]: Advancing to term 4 04:10:04.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.794503 2884 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 4. 04:10:04.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.795557 2791 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 04:10:04.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:04.801221 2788 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:04.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.803424 2925 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 4 election: Requested vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:04.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.804013 2929 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 4 FOLLOWER]: Leader election won for term 4 04:10:04.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.809608 2929 raft_consensus.cc:667] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 4 LEADER]: Becoming Leader. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Running, Role: LEADER 04:10:04.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.810228 2929 consensus_queue.cc:206] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:04.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.819860 2925 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 4 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:04.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.820572 2925 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:04.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.823293 2932 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:10:04.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.830976 2932 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:10:04.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.838747 2932 catalog_manager.cc:1084] Loading token signing keys... 04:10:04.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:04.845377 2932 catalog_manager.cc:4361] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Loaded TSK: 0 04:10:05.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:05.283702 2788 consensus_peers.cc:458] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 -> Peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Couldn't send request to peer 513938f6acf044ed88d2d88de9565cad. Status: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:10:05.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:05.355669 2908 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.984s user 0.342s sys 0.621s 04:10:05.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:05.355939 2908 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.984s user 0.342s sys 0.621s 04:10:05.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.360090 2884 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 4 FOLLOWER]: Refusing update from remote peer 8a3f32f20ddc40d986bae579a55da0f1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch) 04:10:05.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.361991 2929 consensus_queue.cc:984] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s 04:10:05.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.376686 2934 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 4 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:05.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.377429 2934 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:05.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.387909 2934 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:05.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.388563 2934 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:05.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.388490 2929 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:05.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.389127 2929 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:05.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.424769 2908 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:05.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.532068 2908 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.61:35161 04:10:05.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.535178 2908 webserver.cc:374] Webserver started at http://127.2.5.61:45345/ using document root and password file 04:10:05.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.536404 2908 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:05.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.543704 2988 sys_catalog.cc:188] Verifying existing consensus state 04:10:05.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.546728 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 2908 04:10:05.547 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:05.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.547899 2988 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap starting. 04:10:05.557 [DEBUG - New I/O worker #59] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:10:05.557 [DEBUG - New I/O worker #59] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc 04:10:05.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:05 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:05.562 [DEBUG - New I/O worker #59] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:10:05.563 [INFO - New I/O worker #59] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:10:05.563 [DEBUG - New I/O worker #59] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.2.5.60 04:10:05.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:05 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:10:05.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.588346 2988 log.cc:525] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Log is configured to *not* fsync() on all Append() calls 04:10:05.586 [DEBUG - New I/O worker #59] (Negotiator.java:764) Authenticated connection [id: 0xefd0f44b, /127.0.0.1:58418 => /127.2.5.60:39705] using SASL/GSSAPI 04:10:05.591 [INFO - Time-limited test] (MiniKuduCluster.java:439) Running kinit for user test-admin 04:10:05.591 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: kinit { username: "test-admin" } 04:10:05.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.604882 2988 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:05.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.605734 2988 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap complete. 04:10:05.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:05 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:05.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:10:05.648 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:05.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.646087 2988 raft_consensus.cc:340] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:05.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.652565 2988 raft_consensus.cc:705] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513938f6acf044ed88d2d88de9565cad, State: Initialized, Role: FOLLOWER 04:10:05.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.653306 2988 consensus_queue.cc:229] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.655711 2988 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:05.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.656399 2988 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:05.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.657436 2988 sys_catalog.cc:427] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: configured and running, proceeding with master startup. 04:10:05.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.656092 2993 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:05.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.739859 2993 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:05.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.750756 2994 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: acquiring CA information for follower catalog manager: success 04:10:05.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.756351 2994 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:05.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.834703 2965 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 3 FOLLOWER]: Advancing to term 4 04:10:05.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.848520 3001 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:05.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.849334 3001 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:05.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.848980 2993 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 4 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:05.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:05.849658 2993 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:06.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.445292 2929 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:06.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.446346 2929 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:06.649 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.60:39705 04:10:06.649 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:10:06.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.650012 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 2778 04:10:06.674 [DEBUG - New I/O worker #59] (Connection.java:691) [peer master-127.2.5.60:39705(127.2.5.60:39705)] cleaning up while in state READY due to: connection disconnected 04:10:06.675 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:06.676 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.62:35093 04:10:06.676 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:10:06.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.676733 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 2841 04:10:06.698 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:06.698 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.61:35161 04:10:06.698 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:10:06.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.699373 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 2908 04:10:06.719 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:06.719 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.60:39705 04:10:06.719 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:10:06.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.720477 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:06.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:06.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:06.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:06.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:06.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:06.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:06.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:06.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:06.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:06.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:06.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:06.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:06.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:06.867862 3005 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:06.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:06.868213 3005 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.60 04:10:06.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:06.868444 3005 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:06.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:06.879443 3005 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:06.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:06.879590 3005 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:06.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:06.879700 3005 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:06.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:06.879788 3005 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:06.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:06.879901 3005 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:06.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:06.880029 3005 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.60 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.897919 3005 master_main.cc:98] Master server non-default flags: 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:06.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:06.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:06.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.912009 3005 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 140016us 04:10:06.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:06 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608606, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:06.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.914237 3005 init.cc:359] Logged in from keytab as kudu/127.2.5.60@KRBTEST.COM (short username kudu) 04:10:06.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.918045 3005 fs_manager.cc:263] Metadata directory not provided 04:10:06.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.918165 3005 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:06.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.923434 3005 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s 04:10:06.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.924052 3005 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:06.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.924533 3005 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:06.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.931634 3005 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s 04:10:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.931761 3005 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:49 on dist-test-slave-dist-test-slave-fvsv" 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:06.932987 3005 fs_report.cc:352] FS layout report 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/data 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:06.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:07.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:07.590431 3005 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.634s user 0.224s sys 0.389s 04:10:07.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:07.590710 3005 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.634s user 0.224s sys 0.389s 04:10:07.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.682628 3005 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:07.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.748047 3005 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.60:39705 04:10:07.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.750492 3005 webserver.cc:374] Webserver started at http://127.2.5.60:34779/ using document root and password file 04:10:07.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.751511 3005 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:07.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.753227 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3005 04:10:07.753 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:07.753 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.62:35093 04:10:07.753 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:10:07.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.754719 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:07.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:07.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:07.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.764322 3067 sys_catalog.cc:188] Verifying existing consensus state 04:10:07.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.772235 3067 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap starting. 04:10:07.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.833009 3067 log.cc:525] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Log is configured to *not* fsync() on all Append() calls 04:10:07.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.843164 3067 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:07.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.843765 3067 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap complete. 04:10:07.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.863374 3067 raft_consensus.cc:340] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:07.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.868609 3067 raft_consensus.cc:705] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Initialized, Role: FOLLOWER 04:10:07.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.869187 3067 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:07.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.871635 3067 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:07.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.872385 3067 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:07.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.873262 3067 sys_catalog.cc:427] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: configured and running, proceeding with master startup. 04:10:07.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:07.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:07.957413 3068 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:07.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:07.957779 3068 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.62 04:10:07.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:07.958113 3068 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:07.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:07.969369 3068 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:07.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:07.969576 3068 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:07.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:07.969727 3068 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:07.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:07.969871 3068 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:07.970048 3068 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:07.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:07.970221 3068 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.62 04:10:07.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.872004 3070 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:07.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.980461 3070 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:07.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.989396 3071 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: acquiring CA information for follower catalog manager: success 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.990990 3068 master_main.cc:98] Master server non-default flags: 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:07.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:07.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:07.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:07.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:07.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:07.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:07.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:07.992285 3071 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:08.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.005733 3068 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 140516us 04:10:08.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:08 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608608, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:08.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.008116 3068 init.cc:359] Logged in from keytab as kudu/127.2.5.62@KRBTEST.COM (short username kudu) 04:10:08.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.011809 3068 fs_manager.cc:263] Metadata directory not provided 04:10:08.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.011943 3068 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:08.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.017576 3068 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s 04:10:08.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.018204 3068 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:08.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.018710 3068 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:08.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.025621 3068 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.025744 3068 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:46 on dist-test-slave-dist-test-slave-fvsv" 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.027010 3068 fs_report.cc:352] FS layout report 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/data 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:08.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:08.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:08.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:08.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:08.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:08.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:08.845501 3068 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.786s user 0.278s sys 0.500s 04:10:08.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:08.845770 3068 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.786s user 0.278s sys 0.500s 04:10:08.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.876260 3068 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:08.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.986872 3068 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.62:35093 04:10:08.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.989811 3068 webserver.cc:374] Webserver started at http://127.2.5.62:44763/ using document root and password file 04:10:08.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.990942 3068 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:08.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.997968 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3068 04:10:08.998 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:08.998 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.61:35161 04:10:08.998 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:10:08.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:08.999562 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:08.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:08.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:08.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:09.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.005688 3134 sys_catalog.cc:188] Verifying existing consensus state 04:10:09.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.017810 3134 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap starting. 04:10:09.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.098950 3134 log.cc:525] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Log is configured to *not* fsync() on all Append() calls 04:10:09.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.126364 3134 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:09.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.127188 3134 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap complete. 04:10:09.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.158689 3134 raft_consensus.cc:340] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:09.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.166924 3134 raft_consensus.cc:705] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccf046f8f4748bc9dea8828a7a48863, State: Initialized, Role: FOLLOWER 04:10:09.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.167876 3134 consensus_queue.cc:229] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:09.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.171882 3134 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:09.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.172791 3134 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:09.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.174077 3134 sys_catalog.cc:427] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: configured and running, proceeding with master startup. 04:10:09.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:09.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:09.244210 3135 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:09.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:09.244652 3135 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.61 04:10:09.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:09.244989 3135 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:09.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:09.263828 3135 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:09.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:09.264029 3135 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:09.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:09.264170 3135 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:09.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:09.264298 3135 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:09.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:09.264482 3135 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:09.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:09.264681 3135 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.61 04:10:09.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.172268 3137 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:09.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.273535 3137 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:09.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.284960 3138 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: acquiring CA information for follower catalog manager: success 04:10:09.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.287812 3138 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.291748 3135 master_main.cc:98] Master server non-default flags: 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:09.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:09.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:09.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:09.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:09.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:09.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.315368 3135 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 141516us 04:10:09.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:09 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608609, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:09.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.318442 3135 init.cc:359] Logged in from keytab as kudu/127.2.5.61@KRBTEST.COM (short username kudu) 04:10:09.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.332161 3135 fs_manager.cc:263] Metadata directory not provided 04:10:09.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.332625 3135 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:09.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.340967 3135 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s 04:10:09.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.342087 3135 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:09.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.342972 3135 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:09.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.354363 3135 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s 04:10:09.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.354718 3135 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:09.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "513938f6acf044ed88d2d88de9565cad" 04:10:09.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:48 on dist-test-slave-dist-test-slave-fvsv" 04:10:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.356937 3135 fs_report.cc:352] FS layout report 04:10:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/data 04:10:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:09.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:09.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:09.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:09.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:09.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:09.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.399266 3151 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:10:09.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.400240 3151 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:09.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.410116 3151 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:09.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:09.412356 3015 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:09.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.427541 3111 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" is_pre_election: true 04:10:09.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.428675 3111 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 4. 04:10:09.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.430685 3018 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:10:09.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.431953 3151 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 4 FOLLOWER]: Leader pre-election won for term 5 04:10:09.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.432204 3151 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:10:09.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.432437 3151 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 4 FOLLOWER]: Advancing to term 5 04:10:09.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.439561 3151 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:09.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.441737 3111 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:09.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.442219 3111 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 4 FOLLOWER]: Advancing to term 5 04:10:09.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.450505 3111 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 5. 04:10:09.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.451558 3018 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 04:10:09.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:09.456682 3015 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:09.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.458000 3156 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 5 FOLLOWER]: Leader election won for term 5 04:10:09.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.459426 3151 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 5 election: Requested vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:09.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.463728 3156 raft_consensus.cc:667] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 5 LEADER]: Becoming Leader. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Running, Role: LEADER 04:10:09.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.464344 3156 consensus_queue.cc:206] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:09.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.478488 3151 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 5 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:09.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.479439 3151 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:09.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.484272 3159 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:10:09.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.497028 3159 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:10:09.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.511358 3159 catalog_manager.cc:1084] Loading token signing keys... 04:10:09.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.513411 3159 catalog_manager.cc:4361] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Loaded TSK: 0 04:10:09.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:09.902108 3015 consensus_peers.cc:458] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 -> Peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Couldn't send request to peer 513938f6acf044ed88d2d88de9565cad. Status: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:10:09.960 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.960109 3111 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 5 FOLLOWER]: Refusing update from remote peer 8a3f32f20ddc40d986bae579a55da0f1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch) 04:10:09.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.962268 3156 consensus_queue.cc:984] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s 04:10:09.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.976493 3160 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 5 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:09.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.977429 3160 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:09.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.983264 3151 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:09.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.984004 3151 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:09.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.988030 3160 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:09.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:09.989152 3160 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:10.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:10.247277 3135 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.851s user 0.303s sys 0.519s 04:10:10.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:10.247500 3135 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.852s user 0.303s sys 0.519s 04:10:10.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.283919 3135 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:10.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.455287 3135 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.61:35161 04:10:10.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.467579 3135 webserver.cc:374] Webserver started at http://127.2.5.61:45345/ using document root and password file 04:10:10.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.468574 3135 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:10.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.469740 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3135 04:10:10.470 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:10.476 [DEBUG - New I/O worker #60] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:10:10.477 [DEBUG - New I/O worker #60] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc 04:10:10.478 [DEBUG - New I/O worker #60] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:10:10.478 [INFO - New I/O worker #60] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:10:10.479 [DEBUG - New I/O worker #60] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.2.5.60 04:10:10.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.481636 3213 sys_catalog.cc:188] Verifying existing consensus state 04:10:10.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:10 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:10:10.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.495421 3213 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap starting. 04:10:10.505 [DEBUG - New I/O worker #60] (Negotiator.java:764) Authenticated connection [id: 0x75ce4493, /127.0.0.1:58432 => /127.2.5.60:39705] using SASL/GSSAPI 04:10:10.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.554008 3213 log.cc:525] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Log is configured to *not* fsync() on all Append() calls 04:10:10.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.575223 3213 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:10.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.576074 3213 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap complete. 04:10:10.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.607367 3213 raft_consensus.cc:340] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 4 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:10.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.615568 3213 raft_consensus.cc:705] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513938f6acf044ed88d2d88de9565cad, State: Initialized, Role: FOLLOWER 04:10:10.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.616434 3213 consensus_queue.cc:229] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:10.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.619887 3213 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:10.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.620580 3213 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:10.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.621706 3213 sys_catalog.cc:427] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: configured and running, proceeding with master startup. 04:10:10.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.620240 3216 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:10.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.718030 3216 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:10.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.729427 3217 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: acquiring CA information for follower catalog manager: success 04:10:10.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.732383 3217 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:10.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.812357 3190 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 4 FOLLOWER]: Advancing to term 5 04:10:10.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.827934 3222 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:10.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.828766 3222 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:10.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.828387 3216 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 5 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:10.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:10.828991 3216 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:11.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.337365 3156 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:11.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.338084 3156 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:11.509 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.60:39705 04:10:11.509 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:10:11.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.509944 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3005 04:10:11.530 [DEBUG - New I/O worker #60] (Connection.java:691) [peer master-127.2.5.60:39705(127.2.5.60:39705)] cleaning up while in state READY due to: connection disconnected 04:10:11.531 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:11.531 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.62:35093 04:10:11.531 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:10:11.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.532264 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3068 04:10:11.555 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:11.556 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.61:35161 04:10:11.556 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:10:11.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.556653 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3135 04:10:11.576 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:11.577 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.60:39705 04:10:11.577 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:10:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.578368 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:11.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:11.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:11.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:11.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:11.728343 3226 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:11.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:11.728703 3226 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.60 04:10:11.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:11.729030 3226 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:11.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:11.739962 3226 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:11.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:11.740114 3226 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:11.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:11.740260 3226 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:11.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:11.740399 3226 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:11.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:11.740569 3226 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:11.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:11.740739 3226 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.60 04:10:11.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.760676 3226 master_main.cc:98] Master server non-default flags: 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:11.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:11.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:11.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:11.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:11.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:11.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:11.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:11.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:11.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:11.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.775542 3226 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 142516us 04:10:11.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:11 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608611, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:11.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.777822 3226 init.cc:359] Logged in from keytab as kudu/127.2.5.60@KRBTEST.COM (short username kudu) 04:10:11.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.781112 3226 fs_manager.cc:263] Metadata directory not provided 04:10:11.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.781255 3226 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:11.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.786829 3226 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s 04:10:11.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.787539 3226 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:11.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.788077 3226 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:11.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.795264 3226 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:10:11.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.795395 3226 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:11.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:10:11.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:49 on dist-test-slave-dist-test-slave-fvsv" 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:11.796697 3226 fs_report.cc:352] FS layout report 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/data 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:11.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:12.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:12.454685 3226 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.634s user 0.246s sys 0.364s 04:10:12.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:12.454953 3226 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.634s user 0.246s sys 0.364s 04:10:12.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.560273 3226 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:12.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.621400 3226 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.60:39705 04:10:12.624 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.624372 3226 webserver.cc:374] Webserver started at http://127.2.5.60:34779/ using document root and password file 04:10:12.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.625382 3226 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:12.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.632889 3288 sys_catalog.cc:188] Verifying existing consensus state 04:10:12.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.634073 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3226 04:10:12.634 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:12.634 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.62:35093 04:10:12.634 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.635962 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:12.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:12.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:12.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.638644 3288 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap starting. 04:10:12.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.696441 3288 log.cc:525] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Log is configured to *not* fsync() on all Append() calls 04:10:12.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.718986 3288 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:12.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.719861 3288 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap complete. 04:10:12.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.750587 3288 raft_consensus.cc:340] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:12.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.758543 3288 raft_consensus.cc:705] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Initialized, Role: FOLLOWER 04:10:12.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.759373 3288 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:12.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.763872 3288 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:12.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.764595 3288 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:12.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.765674 3288 sys_catalog.cc:427] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: configured and running, proceeding with master startup. 04:10:12.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.764269 3291 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:12.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.865022 3291 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:12.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:12.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:12.866204 3289 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:12.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:12.866520 3289 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.62 04:10:12.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:12.866744 3289 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:12.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.872815 3292 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: acquiring CA information for follower catalog manager: success 04:10:12.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.875696 3292 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:12.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:12.878444 3289 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:12.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:12.878590 3289 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:12.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:12.878780 3289 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:12.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:12.878919 3289 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:12.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:12.879117 3289 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:12.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:12.879309 3289 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.62 04:10:12.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.898764 3289 master_main.cc:98] Master server non-default flags: 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:12.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:12.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:12.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.913386 3289 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 143016us 04:10:12.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:12 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608612, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:12.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.915647 3289 init.cc:359] Logged in from keytab as kudu/127.2.5.62@KRBTEST.COM (short username kudu) 04:10:12.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.918738 3289 fs_manager.cc:263] Metadata directory not provided 04:10:12.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.918860 3289 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.924077 3289 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s 04:10:12.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.924685 3289 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:12.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.925200 3289 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:12.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.932209 3289 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:10:12.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.932330 3289 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:12.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:12.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:46 on dist-test-slave-dist-test-slave-fvsv" 04:10:12.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:12.933575 3289 fs_report.cc:352] FS layout report 04:10:12.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:12.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:12.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:12.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/data 04:10:12.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:12.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:12.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:12.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:12.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:12.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:12.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:12.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:12.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:12.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:13.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:13.740499 3289 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.771s user 0.275s sys 0.490s 04:10:13.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:13.740785 3289 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.772s user 0.275s sys 0.490s 04:10:13.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.784919 3289 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:13.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.901139 3289 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.62:35093 04:10:13.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.907860 3289 webserver.cc:374] Webserver started at http://127.2.5.62:44763/ using document root and password file 04:10:13.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.909130 3289 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:13.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.911231 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3289 04:10:13.911 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:13.911 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.61:35161 04:10:13.911 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.912907 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:13.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:13.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.919164 3355 sys_catalog.cc:188] Verifying existing consensus state 04:10:13.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.924772 3355 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap starting. 04:10:13.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.961557 3355 log.cc:525] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Log is configured to *not* fsync() on all Append() calls 04:10:13.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.971918 3355 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:13.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.972522 3355 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap complete. 04:10:13.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.992080 3355 raft_consensus.cc:340] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:13.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.997108 3355 raft_consensus.cc:705] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccf046f8f4748bc9dea8828a7a48863, State: Initialized, Role: FOLLOWER 04:10:13.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.997680 3355 consensus_queue.cc:229] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:14.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:13.999661 3355 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:14.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.000130 3355 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:14.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.001161 3355 sys_catalog.cc:427] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: configured and running, proceeding with master startup. 04:10:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.000116 3358 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:14.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.070132 3358 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:14.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.100905 3359 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: acquiring CA information for follower catalog manager: success 04:10:14.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.103761 3359 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:14.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:14.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:14.219161 3356 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:14.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:14.219600 3356 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.61 04:10:14.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:14.219938 3356 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:14.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:14.238909 3356 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:14.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:14.239117 3356 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:14.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:14.239280 3356 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:14.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:14.239428 3356 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:14.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:14.239619 3356 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:14.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:14.239814 3356 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.61 04:10:14.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.291568 3356 master_main.cc:98] Master server non-default flags: 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:14.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:14.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:14.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:14.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:14.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:14.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:14.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:14.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.314718 3356 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 144016us 04:10:14.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:14 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608614, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:14.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.317770 3356 init.cc:359] Logged in from keytab as kudu/127.2.5.61@KRBTEST.COM (short username kudu) 04:10:14.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.331936 3356 fs_manager.cc:263] Metadata directory not provided 04:10:14.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.332132 3356 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:14.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.339123 3356 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s 04:10:14.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.339978 3356 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:14.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.340629 3356 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:14.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.351732 3356 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s 04:10:14.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.351903 3356 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:14.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "513938f6acf044ed88d2d88de9565cad" 04:10:14.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:48 on dist-test-slave-dist-test-slave-fvsv" 04:10:14.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.353667 3356 fs_report.cc:352] FS layout report 04:10:14.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:14.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:14.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:14.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/data 04:10:14.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:14.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:14.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:14.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:14.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:14.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:14.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:14.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:14.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:14.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:14.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.607937 3373 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:10:14.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.608458 3373 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:14.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.615121 3373 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:14.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:14.616749 3236 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:14.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.633038 3332 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" is_pre_election: true 04:10:14.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.633810 3332 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 5. 04:10:14.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.635146 3239 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:10:14.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.636252 3373 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 5 FOLLOWER]: Leader pre-election won for term 6 04:10:14.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.636495 3373 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:10:14.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.636720 3373 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 5 FOLLOWER]: Advancing to term 6 04:10:14.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.643904 3373 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:14.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.645869 3332 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:14.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.646308 3332 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 5 FOLLOWER]: Advancing to term 6 04:10:14.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.652464 3332 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 6. 04:10:14.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.653209 3239 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 04:10:14.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.656312 3373 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 6 election: Requested vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:14.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:14.656630 3236 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:14.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.656991 3373 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 6 FOLLOWER]: Leader election won for term 6 04:10:14.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.657369 3373 raft_consensus.cc:667] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 6 LEADER]: Becoming Leader. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Running, Role: LEADER 04:10:14.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.658064 3373 consensus_queue.cc:206] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:14.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.668464 3377 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 6 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:14.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.669309 3377 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:14.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.672078 3379 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:10:14.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.678373 3379 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:10:14.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.685878 3379 catalog_manager.cc:1084] Loading token signing keys... 04:10:14.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:14.690999 3379 catalog_manager.cc:4361] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Loaded TSK: 0 04:10:15.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:15.126967 3236 consensus_peers.cc:458] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 -> Peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Couldn't send request to peer 513938f6acf044ed88d2d88de9565cad. Status: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:10:15.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.156991 3332 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 6 FOLLOWER]: Refusing update from remote peer 8a3f32f20ddc40d986bae579a55da0f1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch) 04:10:15.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.159091 3377 consensus_queue.cc:984] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s 04:10:15.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.165277 3380 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 6 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:15.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.166203 3380 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:15.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.173931 3380 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:15.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.175016 3380 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.178902 3377 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.179584 3377 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:15.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:15.282270 3356 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.890s user 0.001s sys 0.001s 04:10:15.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:15.282524 3356 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.891s user 0.002s sys 0.001s 04:10:15.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.355836 3356 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:15.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.416910 3356 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.61:35161 04:10:15.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.419818 3356 webserver.cc:374] Webserver started at http://127.2.5.61:45345/ using document root and password file 04:10:15.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.420752 3356 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:15.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.426672 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3356 04:10:15.427 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:15.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.428179 3434 sys_catalog.cc:188] Verifying existing consensus state 04:10:15.431 [DEBUG - New I/O worker #61] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:10:15.431 [DEBUG - New I/O worker #61] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc 04:10:15.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.433269 3434 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap starting. 04:10:15.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:15 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:15.443 [DEBUG - New I/O worker #61] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:10:15.443 [INFO - New I/O worker #61] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:10:15.444 [DEBUG - New I/O worker #61] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.2.5.60 04:10:15.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:15 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:10:15.466 [DEBUG - New I/O worker #61] (Negotiator.java:764) Authenticated connection [id: 0x3a59324b, /127.0.0.1:58446 => /127.2.5.60:39705] using SASL/GSSAPI 04:10:15.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.472659 3434 log.cc:525] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Log is configured to *not* fsync() on all Append() calls 04:10:15.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.482993 3434 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:15.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.483659 3434 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap complete. 04:10:15.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.503028 3434 raft_consensus.cc:340] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 5 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:15.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.510536 3434 raft_consensus.cc:705] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513938f6acf044ed88d2d88de9565cad, State: Initialized, Role: FOLLOWER 04:10:15.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.511266 3434 consensus_queue.cc:229] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:15.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.513373 3438 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:15.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.514014 3438 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:15.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:15 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608611, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for kudu/127.2.5.61@KRBTEST.COM 04:10:15.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.514394 3434 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:15.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.514798 3434 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:15.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.515777 3434 sys_catalog.cc:427] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: configured and running, proceeding with master startup. 04:10:15.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.527523 3439 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: acquiring CA information for follower catalog manager: success 04:10:15.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.528571 3411 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 5 FOLLOWER]: Advancing to term 6 04:10:15.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.530308 3439 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:15.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.683272 3443 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:15.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.684631 3443 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:15.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.684136 3438 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 6 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:15.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:15.685106 3438 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:16.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.066669 3382 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:16.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.067538 3382 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:16.471 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.60:39705 04:10:16.472 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:10:16.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.472803 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3226 04:10:16.491 [DEBUG - New I/O worker #61] (Connection.java:691) [peer master-127.2.5.60:39705(127.2.5.60:39705)] cleaning up while in state READY due to: connection disconnected 04:10:16.492 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:16.492 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.62:35093 04:10:16.492 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:10:16.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.492938 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3289 04:10:16.511 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:16.511 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.61:35161 04:10:16.511 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:10:16.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.513118 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3356 04:10:16.531 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:16.531 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.60:39705 04:10:16.531 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.532941 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:16.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:16.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:16.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:16.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:16.685000 3448 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:16.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:16.685333 3448 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.60 04:10:16.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:16.685555 3448 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:16.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:16.696431 3448 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:16.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:16.696575 3448 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:16.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:16.696688 3448 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:16.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:16.696784 3448 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:16.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:16.696898 3448 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:16.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:16.697022 3448 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.60 04:10:16.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.714774 3448 master_main.cc:98] Master server non-default flags: 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:16.715 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:16.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:16.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:16.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:16.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:16.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:16.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:16.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:16.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.728760 3448 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 145016us 04:10:16.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:16 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608616, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:16.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.731009 3448 init.cc:359] Logged in from keytab as kudu/127.2.5.60@KRBTEST.COM (short username kudu) 04:10:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.733994 3448 fs_manager.cc:263] Metadata directory not provided 04:10:16.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.734128 3448 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:16.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.739141 3448 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s 04:10:16.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.739779 3448 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:16.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.740249 3448 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:16.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.747725 3448 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.747859 3448 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:49 on dist-test-slave-dist-test-slave-fvsv" 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:16.749130 3448 fs_report.cc:352] FS layout report 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/data 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:16.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:17.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:17.627393 3448 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.854s user 0.330s sys 0.504s 04:10:17.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:17.627616 3448 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.854s user 0.330s sys 0.504s 04:10:17.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.656610 3448 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:17.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.737108 3448 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.60:39705 04:10:17.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.739766 3448 webserver.cc:374] Webserver started at http://127.2.5.60:34779/ using document root and password file 04:10:17.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.740700 3448 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:17.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.742208 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3448 04:10:17.742 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:17.742 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.62:35093 04:10:17.743 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.743928 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:17.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:17.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:17.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.748989 3510 sys_catalog.cc:188] Verifying existing consensus state 04:10:17.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.754204 3510 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap starting. 04:10:17.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.792881 3510 log.cc:525] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Log is configured to *not* fsync() on all Append() calls 04:10:17.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.806883 3510 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:17.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.807729 3510 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap complete. 04:10:17.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.838171 3510 raft_consensus.cc:340] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:17.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.846190 3510 raft_consensus.cc:705] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Initialized, Role: FOLLOWER 04:10:17.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.847064 3510 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:17.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.850005 3513 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:17.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.850739 3513 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:17.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.851227 3510 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:17.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.851835 3510 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:17.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.852910 3510 sys_catalog.cc:427] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: configured and running, proceeding with master startup. 04:10:17.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.861212 3514 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: acquiring CA information for follower catalog manager: success 04:10:17.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:17.863257 3514 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:17.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:17.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:17.965942 3511 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:17.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:17.966423 3511 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.62 04:10:17.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:17.966804 3511 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:17.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:17.986081 3511 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:17.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:17.986291 3511 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:17.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:17.986467 3511 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:17.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:17.986629 3511 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:17.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:17.986833 3511 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:17.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:17.987063 3511 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.62 04:10:18.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:18.019785 3511 master_main.cc:98] Master server non-default flags: 04:10:18.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:18.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:18.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:18.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:18.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:18.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:18.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:18.043658 3511 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 146016us 04:10:18.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608618, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:18.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:18.046602 3511 init.cc:359] Logged in from keytab as kudu/127.2.5.62@KRBTEST.COM (short username kudu) 04:10:18.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:18.053409 3511 fs_manager.cc:263] Metadata directory not provided 04:10:18.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:18.053589 3511 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:18.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:18.060813 3511 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s 04:10:18.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:18.061681 3511 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:18.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:18.062361 3511 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:18.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:18.076722 3511 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s 04:10:18.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:18.076912 3511 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:18.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:18.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:46 on dist-test-slave-dist-test-slave-fvsv" 04:10:18.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:18.078814 3511 fs_report.cc:352] FS layout report 04:10:18.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:18.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:18.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:18.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/data 04:10:18.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:18.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:18.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:18.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:18.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:18.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:18.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:18.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:18.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:18.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:18.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:18.768057 3511 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.655s user 0.287s sys 0.367s 04:10:18.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:18.768317 3511 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.655s user 0.287s sys 0.367s 04:10:18.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:18.865685 3511 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:19.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.043861 3511 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.62:35093 04:10:19.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.047833 3511 webserver.cc:374] Webserver started at http://127.2.5.62:44763/ using document root and password file 04:10:19.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.049070 3511 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:19.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.054849 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3511 04:10:19.055 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:19.055 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.61:35161 04:10:19.055 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:10:19.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.056496 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:19.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:19.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:19.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:19.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:19.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:19.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:19.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:19.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:19.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:19.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:19.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:19.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.061287 3577 sys_catalog.cc:188] Verifying existing consensus state 04:10:19.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.067842 3578 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:10:19.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.068294 3578 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:19.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.070479 3577 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap starting. 04:10:19.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.070652 3578 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:19.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.080330 3458 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:19.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:19 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608616, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for kudu/127.2.5.62@KRBTEST.COM 04:10:19.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.140508 3577 log.cc:525] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Log is configured to *not* fsync() on all Append() calls 04:10:19.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.155947 3577 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:19.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.156757 3577 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap complete. 04:10:19.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.176470 3577 raft_consensus.cc:340] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:19.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.184854 3577 raft_consensus.cc:705] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccf046f8f4748bc9dea8828a7a48863, State: Initialized, Role: FOLLOWER 04:10:19.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.185760 3577 consensus_queue.cc:229] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:19.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.191929 3577 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:19.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.192661 3577 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:19.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.193730 3577 sys_catalog.cc:427] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: configured and running, proceeding with master startup. 04:10:19.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.263010 3554 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" is_pre_election: true 04:10:19.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.192132 3584 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:19.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.270583 3584 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:19.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.269892 3585 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:10:19.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.270679 3554 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 6. 04:10:19.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.272244 3461 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:10:19.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.272999 3578 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 6 FOLLOWER]: Leader pre-election won for term 7 04:10:19.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.273241 3578 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:10:19.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.273396 3578 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 6 FOLLOWER]: Advancing to term 7 04:10:19.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.277791 3578 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:19.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.279641 3554 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:19.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.280133 3458 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:19.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.280237 3554 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 6 FOLLOWER]: Advancing to term 7 04:10:19.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.280485 3578 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 7 election: Requested vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:19.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.284757 3554 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 7. 04:10:19.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:19.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.302386 3579 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:19.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.302696 3579 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.61 04:10:19.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.302927 3579 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:19.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.317726 3579 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:19.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.317927 3579 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:19.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.318094 3579 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:19.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.318239 3579 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:19.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.318426 3579 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:19.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.318616 3579 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.61 04:10:19.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.350378 3579 master_main.cc:98] Master server non-default flags: 04:10:19.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:19.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:19.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:19.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:19.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:19.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:19.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:19.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:19.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:19.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:19.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:19.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:19.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:19.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:19.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:19.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:19.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:19.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:19.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.369087 3579 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 146516us 04:10:19.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:19 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608619, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:19.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.371771 3579 init.cc:359] Logged in from keytab as kudu/127.2.5.61@KRBTEST.COM (short username kudu) 04:10:19.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.376412 3579 fs_manager.cc:263] Metadata directory not provided 04:10:19.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.376595 3579 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:19.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.285511 3461 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:10:19.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.381513 3461 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.2.5.62:35093, user_credentials={real_user=slave}} blocked reactor thread for 96032.5us 04:10:19.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.382072 3578 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 7 FOLLOWER]: Leader election won for term 7 04:10:19.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.383802 3579 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s 04:10:19.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.383983 3578 raft_consensus.cc:667] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 7 LEADER]: Becoming Leader. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Running, Role: LEADER 04:10:19.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.384706 3579 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:19.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.384599 3578 consensus_queue.cc:206] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:19.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.385313 3579 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:19.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.390296 3593 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 7 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:19.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.391090 3593 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:19.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.393311 3595 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:10:19.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.395208 3579 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s 04:10:19.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.395359 3579 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:19.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "513938f6acf044ed88d2d88de9565cad" 04:10:19.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:48 on dist-test-slave-dist-test-slave-fvsv" 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.396700 3579 fs_report.cc:352] FS layout report 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/data 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:19.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.397565 3595 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:10:19.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.400290 3595 catalog_manager.cc:1084] Loading token signing keys... 04:10:19.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.402201 3595 catalog_manager.cc:4361] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Loaded TSK: 0 04:10:19.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.902606 3554 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 7 FOLLOWER]: Refusing update from remote peer 8a3f32f20ddc40d986bae579a55da0f1: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch) 04:10:19.904 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.904392 3601 consensus_queue.cc:984] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s 04:10:19.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.916718 3602 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 7 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:19.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.917575 3602 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:19.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.933400 3602 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:19.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.934159 3602 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:19.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.945188 3604 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:19.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:19.949995 3604 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:19.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:19.953989 3458 consensus_peers.cc:458] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 -> Peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Couldn't send request to peer 513938f6acf044ed88d2d88de9565cad. Status: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:10:20.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:20.218462 3579 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.791s user 0.001s sys 0.001s 04:10:20.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:20.218675 3579 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.791s user 0.001s sys 0.001s 04:10:20.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.267596 3579 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:20.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.280156 3585 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: acquiring CA information for follower catalog manager: success 04:10:20.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.283439 3585 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:20.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.423583 3579 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.61:35161 04:10:20.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.428328 3579 webserver.cc:374] Webserver started at http://127.2.5.61:45345/ using document root and password file 04:10:20.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.429424 3579 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.433748 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3579 04:10:20.434 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:20.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.439447 3662 sys_catalog.cc:188] Verifying existing consensus state 04:10:20.445 [DEBUG - New I/O worker #62] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:10:20.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.445559 3662 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap starting. 04:10:20.446 [DEBUG - New I/O worker #62] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc 04:10:20.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:20 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:20.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:20 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608616, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for kudu/127.2.5.61@KRBTEST.COM 04:10:20.455 [DEBUG - New I/O worker #62] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:10:20.456 [INFO - New I/O worker #62] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:10:20.458 [DEBUG - New I/O worker #62] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.2.5.60 04:10:20.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:20 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:10:20.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.489287 3662 log.cc:525] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Log is configured to *not* fsync() on all Append() calls 04:10:20.500 [DEBUG - New I/O worker #62] (Negotiator.java:764) Authenticated connection [id: 0xac484daa, /127.0.0.1:58464 => /127.2.5.60:39705] using SASL/GSSAPI 04:10:20.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.502985 3662 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:20.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.503788 3662 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap complete. 04:10:20.505 [INFO - Time-limited test] (MiniKuduCluster.java:439) Running kinit for user test-admin 04:10:20.505 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: kinit { username: "test-admin" } 04:10:20.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.516629 3662 raft_consensus.cc:340] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 6 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:20.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.522336 3662 raft_consensus.cc:705] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513938f6acf044ed88d2d88de9565cad, State: Initialized, Role: FOLLOWER 04:10:20.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.522953 3662 consensus_queue.cc:229] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:20.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.526579 3662 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:20.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.527220 3662 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:20.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.528286 3662 sys_catalog.cc:427] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: configured and running, proceeding with master startup. 04:10:20.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:20 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:20.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:10:20.559 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:20.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.526805 3666 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:20.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.598438 3638 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 6 FOLLOWER]: Advancing to term 7 04:10:20.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.598647 3666 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:20.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.613031 3671 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:20.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.613837 3671 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:20.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.613915 3666 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 7 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:20.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.614527 3666 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:20.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.623380 3667 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: acquiring CA information for follower catalog manager: success 04:10:20.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:20.626260 3667 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.023284 3604 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.026484 3604 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:21.559 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.60:39705 04:10:21.559 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:10:21.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.560423 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3448 04:10:21.585 [DEBUG - New I/O worker #62] (Connection.java:691) [peer master-127.2.5.60:39705(127.2.5.60:39705)] cleaning up while in state READY due to: connection disconnected 04:10:21.587 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:21.587 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.62:35093 04:10:21.587 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:10:21.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.587963 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3511 04:10:21.612 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:21.612 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.61:35161 04:10:21.612 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:10:21.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.613013 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3579 04:10:21.630 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:21.630 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.60:39705 04:10:21.630 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.632140 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:21.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:21.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:21.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:21.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:21.781371 3676 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:21.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:21.781726 3676 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.60 04:10:21.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:21.781956 3676 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:21.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:21.792979 3676 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:21.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:21.793115 3676 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:21.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:21.793224 3676 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:21.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:21.793316 3676 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:21.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:21.793427 3676 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:21.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:21.793547 3676 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.60 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.812279 3676 master_main.cc:98] Master server non-default flags: 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:21.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:21.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:21.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:21.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:21.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.826539 3676 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 147516us 04:10:21.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:21 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:21.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.828847 3676 init.cc:359] Logged in from keytab as kudu/127.2.5.60@KRBTEST.COM (short username kudu) 04:10:21.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.831820 3676 fs_manager.cc:263] Metadata directory not provided 04:10:21.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.831961 3676 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:21.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.836887 3676 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s 04:10:21.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.837582 3676 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:21.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.838129 3676 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:21.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.844768 3676 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s 04:10:21.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.844982 3676 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:21.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:10:21.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:49 on dist-test-slave-dist-test-slave-fvsv" 04:10:21.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:21.846393 3676 fs_report.cc:352] FS layout report 04:10:21.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:21.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:21.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:21.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/data 04:10:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:21.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:22.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:22.498602 3676 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.628s user 0.262s sys 0.339s 04:10:22.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:22.498800 3676 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.628s user 0.262s sys 0.339s 04:10:22.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.543437 3676 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:22.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.629695 3676 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.60:39705 04:10:22.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.632017 3676 webserver.cc:374] Webserver started at http://127.2.5.60:34779/ using document root and password file 04:10:22.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.632959 3676 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:22.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.637349 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3676 04:10:22.637 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:22.637 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.62:35093 04:10:22.638 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.639221 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:22.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:22.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:22.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.641212 3738 sys_catalog.cc:188] Verifying existing consensus state 04:10:22.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.645967 3738 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap starting. 04:10:22.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.708933 3738 log.cc:525] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Log is configured to *not* fsync() on all Append() calls 04:10:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.724910 3738 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:22.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.725592 3738 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap complete. 04:10:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.745573 3738 raft_consensus.cc:340] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:22.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.750766 3738 raft_consensus.cc:705] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Initialized, Role: FOLLOWER 04:10:22.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.751482 3738 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:22.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.753865 3738 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:22.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.754663 3738 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:22.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.755780 3738 sys_catalog.cc:427] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: configured and running, proceeding with master startup. 04:10:22.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:22.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:22.833762 3739 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:22.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:22.834133 3739 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.62 04:10:22.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:22.834461 3739 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:22.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:22.846149 3739 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:22.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:22.846316 3739 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:22.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:22.846462 3739 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:22.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:22.846596 3739 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:22.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:22.846776 3739 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:22.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:22.846951 3739 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.62 04:10:22.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.867521 3739 master_main.cc:98] Master server non-default flags: 04:10:22.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:22.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:22.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:22.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.754231 3741 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:22.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.876281 3741 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:22.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.882761 3739 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 148016us 04:10:22.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:22 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608622, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:22.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.885054 3739 init.cc:359] Logged in from keytab as kudu/127.2.5.62@KRBTEST.COM (short username kudu) 04:10:22.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.886648 3742 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: acquiring CA information for follower catalog manager: success 04:10:22.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.892702 3742 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:22.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.893642 3739 fs_manager.cc:263] Metadata directory not provided 04:10:22.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.893772 3739 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:22.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.900421 3739 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s 04:10:22.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.901089 3739 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:22.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.901599 3739 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:22.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.912237 3739 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s 04:10:22.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.912369 3739 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:22.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:22.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:46 on dist-test-slave-dist-test-slave-fvsv" 04:10:22.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:22.913671 3739 fs_report.cc:352] FS layout report 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/data 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:22.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:23.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:23.699079 3739 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.754s user 0.312s sys 0.440s 04:10:23.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:23.699359 3739 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.754s user 0.313s sys 0.440s 04:10:23.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:23.748420 3739 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:23.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:23.923537 3739 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.62:35093 04:10:23.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:23.927511 3739 webserver.cc:374] Webserver started at http://127.2.5.62:44763/ using document root and password file 04:10:23.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:23.928527 3739 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:23.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:23.936630 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3739 04:10:23.937 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:23.937 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.61:35161 04:10:23.937 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:10:23.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:23.938326 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:23.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:23.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:23.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:23.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:23.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:23.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:23.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:23.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:23.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:23.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:23.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:23.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:23.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:23.941054 3805 sys_catalog.cc:188] Verifying existing consensus state 04:10:23.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:23.952980 3805 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap starting. 04:10:24.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.026412 3805 log.cc:525] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Log is configured to *not* fsync() on all Append() calls 04:10:24.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.048413 3805 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:24.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.049219 3805 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap complete. 04:10:24.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.080662 3805 raft_consensus.cc:340] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 7 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:24.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.089131 3805 raft_consensus.cc:705] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccf046f8f4748bc9dea8828a7a48863, State: Initialized, Role: FOLLOWER 04:10:24.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.090176 3805 consensus_queue.cc:229] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:24.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.093133 3808 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:24.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.093829 3808 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:24.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.094264 3805 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:24.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.094833 3805 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:24.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.095966 3805 sys_catalog.cc:427] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: configured and running, proceeding with master startup. 04:10:24.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.097894 3809 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:10:24.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:24.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:24.148792 3806 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:24.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:24.149125 3806 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.61 04:10:24.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:24.149350 3806 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:24.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:24.160176 3806 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:24.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:24.160313 3806 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:24.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:24.160420 3806 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:24.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:24.160516 3806 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:24.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:24.160624 3806 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:24.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:24.160758 3806 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.61 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.187767 3806 master_main.cc:98] Master server non-default flags: 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:24.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.188567 3814 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:10:24.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.188995 3814 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:24.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.191607 3814 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:24.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:24.204181 3686 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:24.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.211364 3806 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 149016us 04:10:24.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:24 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for kudu/127.2.5.62@KRBTEST.COM 04:10:24.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:24 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:24.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.214618 3806 init.cc:359] Logged in from keytab as kudu/127.2.5.61@KRBTEST.COM (short username kudu) 04:10:24.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.224130 3806 fs_manager.cc:263] Metadata directory not provided 04:10:24.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.224309 3806 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:24.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.224463 3782 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" is_pre_election: true 04:10:24.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.225107 3782 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 7. 04:10:24.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.226397 3689 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:10:24.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.227193 3814 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 7 FOLLOWER]: Leader pre-election won for term 8 04:10:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.227424 3814 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:10:24.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.227581 3814 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 7 FOLLOWER]: Advancing to term 8 04:10:24.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.231592 3806 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s 04:10:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.232262 3806 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.232156 3814 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:24.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.232769 3806 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:24.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.233355 3814 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 8 election: Requested vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:24.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.234009 3782 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:24.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.234448 3782 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 7 FOLLOWER]: Advancing to term 8 04:10:24.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:24.234724 3686 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:24.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.240911 3782 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 8. 04:10:24.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.241492 3806 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s 04:10:24.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.241753 3689 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:10:24.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.242436 3814 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 8 FOLLOWER]: Leader election won for term 8 04:10:24.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.241660 3806 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:24.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "513938f6acf044ed88d2d88de9565cad" 04:10:24.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:48 on dist-test-slave-dist-test-slave-fvsv" 04:10:24.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.243556 3806 fs_report.cc:352] FS layout report 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/data 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:24.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:24.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.251792 3814 raft_consensus.cc:667] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 8 LEADER]: Becoming Leader. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Running, Role: LEADER 04:10:24.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.252717 3814 consensus_queue.cc:206] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:24.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.260658 3822 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 8 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:24.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.261976 3822 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:24.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.267868 3824 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:10:24.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.270627 3824 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:10:24.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.272441 3824 catalog_manager.cc:1084] Loading token signing keys... 04:10:24.273 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.273747 3824 catalog_manager.cc:4361] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Loaded TSK: 0 04:10:24.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:24.668838 3686 consensus_peers.cc:458] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 -> Peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Couldn't send request to peer 513938f6acf044ed88d2d88de9565cad. Status: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:10:24.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.737718 3782 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 8 FOLLOWER]: Refusing update from remote peer 8a3f32f20ddc40d986bae579a55da0f1: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch) 04:10:24.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.739014 3822 consensus_queue.cc:984] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s 04:10:24.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.756737 3829 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 8 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:24.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.757800 3829 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:24.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.769398 3829 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:24.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.770205 3829 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:24.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.768826 3814 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:24.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:24.770771 3814 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:25.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:25.148195 3806 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.866s user 0.324s sys 0.507s 04:10:25.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:25.148475 3806 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.866s user 0.324s sys 0.507s 04:10:25.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.200919 3806 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:25.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.107012 3809 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: acquiring CA information for follower catalog manager: success 04:10:25.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.247617 3809 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:25.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.387540 3806 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.61:35161 04:10:25.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.391513 3806 webserver.cc:374] Webserver started at http://127.2.5.61:45345/ using document root and password file 04:10:25.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.392486 3806 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:25.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.400614 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3806 04:10:25.400 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:25.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.406611 3884 sys_catalog.cc:188] Verifying existing consensus state 04:10:25.409 [DEBUG - New I/O worker #55] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:10:25.409 [DEBUG - New I/O worker #55] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc 04:10:25.410 [DEBUG - New I/O worker #55] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:10:25.411 [INFO - New I/O worker #55] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:10:25.411 [DEBUG - New I/O worker #55] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.2.5.60 04:10:25.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:25 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:10:25.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.423825 3884 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap starting. 04:10:25.429 [DEBUG - New I/O worker #55] (Negotiator.java:764) Authenticated connection [id: 0x7e5a2ce5, /127.0.0.1:58476 => /127.2.5.60:39705] using SASL/GSSAPI 04:10:25.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.477159 3884 log.cc:525] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Log is configured to *not* fsync() on all Append() calls 04:10:25.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.496922 3884 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:25.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.497555 3884 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap complete. 04:10:25.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.517246 3884 raft_consensus.cc:340] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 7 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:25.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.522495 3884 raft_consensus.cc:705] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 7 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513938f6acf044ed88d2d88de9565cad, State: Initialized, Role: FOLLOWER 04:10:25.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.523131 3884 consensus_queue.cc:229] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:25.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.527603 3884 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:25.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.528075 3884 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:25.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.528940 3884 sys_catalog.cc:427] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: configured and running, proceeding with master startup. 04:10:25.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.527927 3887 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:25.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.629550 3887 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:25.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.641289 3888 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: acquiring CA information for follower catalog manager: success 04:10:25.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.644222 3888 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:25.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.795653 3861 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 7 FOLLOWER]: Advancing to term 8 04:10:25.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.816077 3894 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:25.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.817240 3894 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:25.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.816836 3887 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 8 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:25.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:25.817505 3887 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:26.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.216657 3822 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:26.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.218072 3822 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:26.433 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.60:39705 04:10:26.433 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:10:26.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.433889 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3676 04:10:26.454 [DEBUG - New I/O worker #55] (Connection.java:691) [peer master-127.2.5.60:39705(127.2.5.60:39705)] cleaning up while in state READY due to: connection disconnected 04:10:26.463 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:26.464 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.62:35093 04:10:26.464 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:10:26.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.464792 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3739 04:10:26.484 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:26.484 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.61:35161 04:10:26.484 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:10:26.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.485296 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3806 04:10:26.501 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:26.501 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.60:39705 04:10:26.501 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:10:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.502245 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:26.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:26.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:26.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:26.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:26.653928 3899 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:26.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:26.654281 3899 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.60 04:10:26.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:26.654507 3899 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:26.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:26.665477 3899 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:26.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:26.665632 3899 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:26.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:26.665743 3899 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:26.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:26.665833 3899 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:26.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:26.665956 3899 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:26.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:26.666079 3899 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.60 04:10:26.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.685297 3899 master_main.cc:98] Master server non-default flags: 04:10:26.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:26.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:26.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:26.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:26.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:26.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:26.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:26.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:26.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:26.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:26.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:26.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.699604 3899 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 150016us 04:10:26.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:26 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:26.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.701907 3899 init.cc:359] Logged in from keytab as kudu/127.2.5.60@KRBTEST.COM (short username kudu) 04:10:26.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.705704 3899 fs_manager.cc:263] Metadata directory not provided 04:10:26.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.705837 3899 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:26.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.710889 3899 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s 04:10:26.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.711572 3899 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:26.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.712082 3899 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:26.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.718822 3899 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.719102 3899 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:49 on dist-test-slave-dist-test-slave-fvsv" 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:26.720381 3899 fs_report.cc:352] FS layout report 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/data 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:26.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:26.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:26.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:26.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:27.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:27.380615 3899 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.636s user 0.227s sys 0.384s 04:10:27.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:27.380854 3899 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.637s user 0.227s sys 0.384s 04:10:27.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.493294 3899 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:27.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.549000 3899 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.60:39705 04:10:27.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.551383 3899 webserver.cc:374] Webserver started at http://127.2.5.60:34779/ using document root and password file 04:10:27.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.552310 3899 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:27.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.558900 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3899 04:10:27.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.559207 3961 sys_catalog.cc:188] Verifying existing consensus state 04:10:27.559 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:27.559 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.62:35093 04:10:27.559 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:10:27.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.560621 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:27.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:27.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:27.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:27.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:27.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:27.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:27.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:27.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:27.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:27.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.563189 3961 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap starting. 04:10:27.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.619544 3961 log.cc:525] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Log is configured to *not* fsync() on all Append() calls 04:10:27.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.637742 3961 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:27.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.638417 3961 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap complete. 04:10:27.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.658687 3961 raft_consensus.cc:340] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:27.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.663988 3961 raft_consensus.cc:705] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Initialized, Role: FOLLOWER 04:10:27.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.664733 3961 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:27.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.667572 3961 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:27.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.668102 3961 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:27.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.669138 3961 sys_catalog.cc:427] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: configured and running, proceeding with master startup. 04:10:27.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.667898 3964 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:27.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.740051 3964 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:27.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.753396 3965 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: acquiring CA information for follower catalog manager: success 04:10:27.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.756382 3965 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:27.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:27.777685 3962 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:27.778031 3962 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.62 04:10:27.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:27.778265 3962 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:27.789 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:27.789780 3962 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:27.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:27.789927 3962 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:27.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:27.790035 3962 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:27.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:27.790133 3962 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:27.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:27.790244 3962 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:27.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:27.790362 3962 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.62 04:10:27.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.808936 3962 master_main.cc:98] Master server non-default flags: 04:10:27.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:27.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:27.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:27.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:27.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:27.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:27.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:27.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:27.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:27.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:27.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:27.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:27.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:27.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:27.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:27.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:27.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:27.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:27.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.826272 3962 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 150516us 04:10:27.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:27 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608627, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:27.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.828567 3962 init.cc:359] Logged in from keytab as kudu/127.2.5.62@KRBTEST.COM (short username kudu) 04:10:27.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.841471 3962 fs_manager.cc:263] Metadata directory not provided 04:10:27.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.841650 3962 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:27.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.846995 3962 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s 04:10:27.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.847687 3962 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:27.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.848203 3962 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:27.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.854970 3962 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:10:27.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.855142 3962 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:27.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:27.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:46 on dist-test-slave-dist-test-slave-fvsv" 04:10:27.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:27.856468 3962 fs_report.cc:352] FS layout report 04:10:27.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:27.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:27.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:27.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/data 04:10:27.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:27.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:27.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:27.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:27.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:27.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:27.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:27.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:27.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:27.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:28.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:28.649278 3962 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.762s user 0.317s sys 0.403s 04:10:28.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:28.649560 3962 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.762s user 0.317s sys 0.403s 04:10:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.704478 3962 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:28.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.841598 3962 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.62:35093 04:10:28.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.844020 3962 webserver.cc:374] Webserver started at http://127.2.5.62:44763/ using document root and password file 04:10:28.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.845001 3962 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:28.850 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.850493 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 3962 04:10:28.850 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:28.851 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.61:35161 04:10:28.851 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:10:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.852375 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:28.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:28.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:28.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:28.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:28.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:28.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:28.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:28.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:28.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:28.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.853121 4028 sys_catalog.cc:188] Verifying existing consensus state 04:10:28.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.857384 4028 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap starting. 04:10:28.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.896415 4028 log.cc:525] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Log is configured to *not* fsync() on all Append() calls 04:10:28.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.911861 4028 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:28.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.912688 4028 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap complete. 04:10:28.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.943509 4028 raft_consensus.cc:340] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:28.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.951464 4028 raft_consensus.cc:705] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccf046f8f4748bc9dea8828a7a48863, State: Initialized, Role: FOLLOWER 04:10:28.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.952255 4028 consensus_queue.cc:229] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:28.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.955901 4028 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:28.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.956818 4028 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:28.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.958173 4028 sys_catalog.cc:427] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: configured and running, proceeding with master startup. 04:10:29.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:28.956221 4031 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:29.064 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.063577 4031 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:29.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.077463 4032 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: acquiring CA information for follower catalog manager: success 04:10:29.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.084887 4032 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:29.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:29.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.096843 4029 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:29.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.097225 4029 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.61 04:10:29.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.097559 4029 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:29.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.109084 4029 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:29.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.109235 4029 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:29.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.109380 4029 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:29.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.109529 4029 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:29.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.109692 4029 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:29.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.109863 4029 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.61 04:10:29.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.140707 4029 master_main.cc:98] Master server non-default flags: 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:29.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:29.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:29.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:29.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:29.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:29.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:29.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:29.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:29.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:29.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:29.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:29.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:29.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:29.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:29.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:29.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:29.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:29.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:29.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:29.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.156659 4029 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 151516us 04:10:29.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:29 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608629, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:29.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.159072 4029 init.cc:359] Logged in from keytab as kudu/127.2.5.61@KRBTEST.COM (short username kudu) 04:10:29.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.167939 4029 fs_manager.cc:263] Metadata directory not provided 04:10:29.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.168099 4029 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:29.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.176895 4029 fs_manager.cc:399] Time spent opening directory manager: real 0.007s user 0.003s sys 0.003s 04:10:29.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.177625 4029 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:29.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.178189 4029 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:29.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.185153 4029 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:10:29.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.185292 4029 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:29.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "513938f6acf044ed88d2d88de9565cad" 04:10:29.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:48 on dist-test-slave-dist-test-slave-fvsv" 04:10:29.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.186657 4029 fs_report.cc:352] FS layout report 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/data 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:29.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:29.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.280316 4046 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:10:29.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.280925 4046 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:29.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.288077 4046 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:29.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.293920 3909 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:29.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.327309 4005 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" is_pre_election: true 04:10:29.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.328168 4005 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 8. 04:10:29.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.329804 3912 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:10:29.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.331138 4046 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 8 FOLLOWER]: Leader pre-election won for term 9 04:10:29.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.331415 4046 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:10:29.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.331655 4046 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 8 FOLLOWER]: Advancing to term 9 04:10:29.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.338750 4046 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:29.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.341265 4005 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:29.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.341822 4005 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 8 FOLLOWER]: Advancing to term 9 04:10:29.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.348709 4005 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 9. 04:10:29.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.349649 3912 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 04:10:29.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.362304 3909 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:29.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.363471 4050 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 9 FOLLOWER]: Leader election won for term 9 04:10:29.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.360776 4046 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 9 election: Requested vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:29.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.377728 4050 raft_consensus.cc:667] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 9 LEADER]: Becoming Leader. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Running, Role: LEADER 04:10:29.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.378582 4050 consensus_queue.cc:206] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:29.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.390458 4046 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 9 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:29.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.391325 4046 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:29.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.400102 4053 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:10:29.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.408967 4053 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:10:29.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.412096 4053 catalog_manager.cc:1084] Loading token signing keys... 04:10:29.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:29.414275 4053 catalog_manager.cc:4361] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Loaded TSK: 0 04:10:29.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.948354 4029 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.725s user 0.000s sys 0.002s 04:10:29.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.948657 4029 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.725s user 0.000s sys 0.002s 04:10:29.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:29.972293 3909 consensus_peers.cc:458] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 -> Peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Couldn't send request to peer 513938f6acf044ed88d2d88de9565cad. Status: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:10:30.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.006700 4005 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 9 FOLLOWER]: Refusing update from remote peer 8a3f32f20ddc40d986bae579a55da0f1: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 9 index: 11. (index mismatch) 04:10:30.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.009752 4055 consensus_queue.cc:984] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s 04:10:30.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.018357 4029 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:30.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.021513 4060 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 9 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:30.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.022418 4060 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:30.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.028213 4055 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:30.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.028903 4055 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:30.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.032121 4060 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:30.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.032909 4060 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:30.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.117303 4029 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.61:35161 04:10:30.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.119716 4029 webserver.cc:374] Webserver started at http://127.2.5.61:45345/ using document root and password file 04:10:30.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.120677 4029 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:30.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.126430 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4029 04:10:30.126 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:30.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.128252 4111 sys_catalog.cc:188] Verifying existing consensus state 04:10:30.132 [DEBUG - New I/O worker #56] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:10:30.133 [DEBUG - New I/O worker #56] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc 04:10:30.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.133865 4111 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap starting. 04:10:30.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:30.138 [DEBUG - New I/O worker #56] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:10:30.138 [INFO - New I/O worker #56] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:10:30.140 [DEBUG - New I/O worker #56] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.2.5.60 04:10:30.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:10:30.156 [DEBUG - New I/O worker #56] (Negotiator.java:764) Authenticated connection [id: 0x199856b6, /127.0.0.1:58488 => /127.2.5.60:39705] using SASL/GSSAPI 04:10:30.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.176849 4111 log.cc:525] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Log is configured to *not* fsync() on all Append() calls 04:10:30.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.188848 4111 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:30.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.189445 4111 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap complete. 04:10:30.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.209296 4111 raft_consensus.cc:340] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 8 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:30.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.214409 4111 raft_consensus.cc:705] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513938f6acf044ed88d2d88de9565cad, State: Initialized, Role: FOLLOWER 04:10:30.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.214973 4111 consensus_queue.cc:229] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:30.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.217178 4114 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:30.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.217674 4114 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:30.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.217967 4111 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:30.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.218349 4111 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:30.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.219133 4111 sys_catalog.cc:427] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: configured and running, proceeding with master startup. 04:10:30.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.220495 4115 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:10:30.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608626, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for kudu/127.2.5.61@KRBTEST.COM 04:10:30.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.392796 4088 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [term 8 FOLLOWER]: Advancing to term 9 04:10:30.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.408116 4119 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:30.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.408828 4119 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:30.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.408845 4114 sys_catalog.cc:339] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 9 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:30.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.521670 4114 sys_catalog.cc:342] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad [sys.catalog]: This master's current role is: FOLLOWER 04:10:30.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.825796 4059 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:30.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:30.826387 4059 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:31.160 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.60:39705 04:10:31.161 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:10:31.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.161728 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3899 04:10:31.181 [DEBUG - New I/O worker #56] (Connection.java:691) [peer master-127.2.5.60:39705(127.2.5.60:39705)] cleaning up while in state READY due to: connection disconnected 04:10:31.182 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:31.182 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.62:35093 04:10:31.182 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:10:31.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.182993 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 3962 04:10:31.203 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:31.204 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.2.5.61:35161 04:10:31.204 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:10:31.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.204507 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4029 04:10:31.218 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:31.218 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.60:39705 04:10:31.218 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.219806 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:31.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:31.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:31.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:31.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:31.369830 4123 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:31.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:31.370149 4123 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.60 04:10:31.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:31.370396 4123 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:31.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:31.381299 4123 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:31.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:31.381446 4123 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:31.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:31.381553 4123 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:31.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:31.381642 4123 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:31.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:31.381786 4123 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:31.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:31.381918 4123 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.60 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.400251 4123 master_main.cc:98] Master server non-default flags: 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.60:39705 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.60.keytab 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.60 04:10:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.60 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=34779 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/logs 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:31.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:31.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.414396 4123 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 152516us 04:10:31.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:31 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608631, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:31.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.416894 4123 init.cc:359] Logged in from keytab as kudu/127.2.5.60@KRBTEST.COM (short username kudu) 04:10:31.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.419695 4123 fs_manager.cc:263] Metadata directory not provided 04:10:31.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.419819 4123 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:31.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.424685 4123 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s 04:10:31.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.425379 4123 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:31.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.425885 4123 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:31.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.432626 4123 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s 04:10:31.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.432749 4123 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "8a3f32f20ddc40d986bae579a55da0f1" 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:49 on dist-test-slave-dist-test-slave-fvsv" 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:31.434000 4123 fs_report.cc:352] FS layout report 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/wal 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/data 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:31.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:32.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:32.097718 4123 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.640s user 0.225s sys 0.389s 04:10:32.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:32.097952 4123 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.641s user 0.225s sys 0.389s 04:10:32.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.167842 4123 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:32.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.236914 4123 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.60:39705 04:10:32.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.239292 4123 webserver.cc:374] Webserver started at http://127.2.5.60:34779/ using document root and password file 04:10:32.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.240222 4123 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-2/data/info.pb 04:10:32.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.246331 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4123 04:10:32.246 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:32.246 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.62:35093 04:10:32.246 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:10:32.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.247457 4185 sys_catalog.cc:188] Verifying existing consensus state 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.248070 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:32.248 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:32.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:32.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:32.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:32.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:32.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:32.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:32.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:32.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:32.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:32.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.252718 4185 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap starting. 04:10:32.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.310297 4185 log.cc:525] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Log is configured to *not* fsync() on all Append() calls 04:10:32.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.333679 4185 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:32.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.334487 4185 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Bootstrap complete. 04:10:32.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.364892 4185 raft_consensus.cc:340] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:32.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.372910 4185 raft_consensus.cc:705] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Initialized, Role: FOLLOWER 04:10:32.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.373699 4185 consensus_queue.cc:229] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:32.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.387960 4185 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:32.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.388926 4185 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:32.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.390177 4185 sys_catalog.cc:427] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: configured and running, proceeding with master startup. 04:10:32.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:32.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:32.471819 4186 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:32.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:32.472188 4186 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.62 04:10:32.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:32.472524 4186 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:32.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:32.483575 4186 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:32.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:32.483784 4186 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:32.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:32.483929 4186 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:32.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:32.484067 4186 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:32.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:32.484251 4186 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:32.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:32.484423 4186 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.62 04:10:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.388280 4188 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:32.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.491801 4188 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: FOLLOWER 04:10:32.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.502786 4189 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: acquiring CA information for follower catalog manager: success 04:10:32.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.504775 4189 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:32.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.515573 4186 master_main.cc:98] Master server non-default flags: 04:10:32.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data 04:10:32.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:32.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.62:35093 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.62.keytab 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.62 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.62 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=44763 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:32.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:32.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:32.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:32.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/logs 04:10:32.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:32.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:32.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:32.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:32.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:32.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:32.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:32.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:32.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:32.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.530133 4186 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 153016us 04:10:32.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:32 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608632, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:32.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.532558 4186 init.cc:359] Logged in from keytab as kudu/127.2.5.62@KRBTEST.COM (short username kudu) 04:10:32.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.547770 4186 fs_manager.cc:263] Metadata directory not provided 04:10:32.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.547919 4186 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:32.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.556481 4186 fs_manager.cc:399] Time spent opening directory manager: real 0.007s user 0.008s sys 0.000s 04:10:32.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.557181 4186 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:32.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.557749 4186 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:32.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.564697 4186 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.564838 4186 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:46 on dist-test-slave-dist-test-slave-fvsv" 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:32.566186 4186 fs_report.cc:352] FS layout report 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/wal 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/data 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:32.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:32.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:32.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:32.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:32.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:32.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:33.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:33.221797 4186 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.631s user 0.199s sys 0.430s 04:10:33.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:33.222064 4186 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.631s user 0.199s sys 0.430s 04:10:33.280 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.280447 4186 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:33.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.451975 4186 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.62:35093 04:10:33.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.455132 4186 webserver.cc:374] Webserver started at http://127.2.5.62:44763/ using document root and password file 04:10:33.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.456372 4186 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-0/data/info.pb 04:10:33.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.456946 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4186 04:10:33.457 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:33.457 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.2.5.61:35161 04:10:33.457 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:10:33.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.458564 2068 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:33.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:33.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:33.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:33.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:33.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:33.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:33.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:33.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:33.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:33.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:33.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:33.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:33.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:33.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.465493 4252 sys_catalog.cc:188] Verifying existing consensus state 04:10:33.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.469743 4252 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap starting. 04:10:33.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.508497 4252 log.cc:525] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Log is configured to *not* fsync() on all Append() calls 04:10:33.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.525074 4252 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:33.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.525907 4252 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: Bootstrap complete. 04:10:33.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.556113 4252 raft_consensus.cc:340] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 9 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:33.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.564035 4252 raft_consensus.cc:705] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ccf046f8f4748bc9dea8828a7a48863, State: Initialized, Role: FOLLOWER 04:10:33.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.564818 4252 consensus_queue.cc:229] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:33.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.567589 4252 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:33.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.568387 4252 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:33.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.569689 4252 sys_catalog.cc:427] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: configured and running, proceeding with master startup. 04:10:33.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.567904 4255 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:33.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.665961 4255 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:33.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.675890 4256 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: acquiring CA information for follower catalog manager: success 04:10:33.679 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.678983 4256 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:33.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:33.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:33.682472 4253 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:33.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:33.683086 4253 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.2.5.61 04:10:33.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:33.683393 4253 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:33.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:33.695017 4253 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:33.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:33.695205 4253 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:33.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:33.695365 4253 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:33.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:33.695510 4253 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:33.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:33.695681 4253 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:33.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:33.695858 4253 flags.cc:404] Enabled experimental flag: --principal=kudu/127.2.5.61 04:10:33.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.723673 4253 master_main.cc:98] Master server non-default flags: 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.2.5.62:35093,127.2.5.61:35161,127.2.5.60:39705 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.2.5.61:35161 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/kudu_127.2.5.61.keytab 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.2.5.61 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:33.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.2.5.61 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=45345 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/logs 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:33.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:33.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.739375 4253 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 153516us 04:10:33.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:33 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608633, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.5.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:33.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.741775 4253 init.cc:359] Logged in from keytab as kudu/127.2.5.61@KRBTEST.COM (short username kudu) 04:10:33.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.745191 4253 fs_manager.cc:263] Metadata directory not provided 04:10:33.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.745405 4253 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:33.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.750463 4253 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s 04:10:33.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.751197 4253 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:33.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.751719 4253 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:33.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.758652 4253 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:10:33.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.758785 4253 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:33.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "513938f6acf044ed88d2d88de9565cad" 04:10:33.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:09:48 on dist-test-slave-dist-test-slave-fvsv" 04:10:33.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.760097 4253 fs_report.cc:352] FS layout report 04:10:33.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:33.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:33.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/wal 04:10:33.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/data 04:10:33.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:33.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:33.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:33.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:33.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:33.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:33.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:33.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:33.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:33.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:33.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.979952 4270 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:10:33.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.980430 4270 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:33.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:33.987627 4270 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:33.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:33.995394 4133 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:34.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.018884 4229 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" is_pre_election: true 04:10:34.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.019842 4229 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 9. 04:10:34.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.022049 4136 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:10:34.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.023174 4270 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 9 FOLLOWER]: Leader pre-election won for term 10 04:10:34.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.023443 4270 raft_consensus.cc:472] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:10:34.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.023671 4270 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 9 FOLLOWER]: Advancing to term 10 04:10:34.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.030844 4270 raft_consensus.cc:494] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:34.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.033203 4229 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a3f32f20ddc40d986bae579a55da0f1" candidate_term: 10 candidate_status { last_received { term: 9 index: 11 } } ignore_live_leader: false dest_uuid: "7ccf046f8f4748bc9dea8828a7a48863" 04:10:34.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.033694 4229 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 9 FOLLOWER]: Advancing to term 10 04:10:34.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:34.041004 4133 leader_election.cc:341] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111) 04:10:34.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.042680 4229 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a3f32f20ddc40d986bae579a55da0f1 in term 10. 04:10:34.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.043104 4270 leader_election.cc:296] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 10 election: Requested vote from peers 7ccf046f8f4748bc9dea8828a7a48863 (127.2.5.62:35093), 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161) 04:10:34.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.044474 4136 leader_election.cc:310] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7ccf046f8f4748bc9dea8828a7a48863, 8a3f32f20ddc40d986bae579a55da0f1; no voters: 513938f6acf044ed88d2d88de9565cad 04:10:34.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.045605 4270 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 10 FOLLOWER]: Leader election won for term 10 04:10:34.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.047847 4270 raft_consensus.cc:667] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [term 10 LEADER]: Becoming Leader. State: Replica: 8a3f32f20ddc40d986bae579a55da0f1, State: Running, Role: LEADER 04:10:34.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.048542 4270 consensus_queue.cc:206] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 9.11, Last appended by leader: 11, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } 04:10:34.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.054740 4274 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 10 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:34.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.055778 4274 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:34.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.068478 4276 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:10:34.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.077836 4276 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:10:34.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.088336 4276 catalog_manager.cc:1084] Loading token signing keys... 04:10:34.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.090890 4276 catalog_manager.cc:4361] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1: Loaded TSK: 0 04:10:34.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:34.466292 4253 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.676s user 0.001s sys 0.001s 04:10:34.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:34.466485 4253 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.676s user 0.001s sys 0.001s 04:10:34.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.512279 4253 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:34.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.603610 4229 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [term 10 FOLLOWER]: Refusing update from remote peer 8a3f32f20ddc40d986bae579a55da0f1: Log matching property violated. Preceding OpId in replica: term: 9 index: 11. Preceding OpId from leader: term: 10 index: 12. (index mismatch) 04:10:34.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.605727 4303 consensus_queue.cc:984] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s 04:10:34.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.616665 4305 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3f32f20ddc40d986bae579a55da0f1. Latest consensus state: current_term: 10 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:34.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.617506 4305 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:34.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.631919 4305 sys_catalog.cc:339] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:34.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:34.641829 4133 consensus_peers.cc:458] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 -> Peer 513938f6acf044ed88d2d88de9565cad (127.2.5.61:35161): Couldn't send request to peer 513938f6acf044ed88d2d88de9565cad. Status: Network error: Client connection negotiation failed: client connection to 127.2.5.61:35161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:10:34.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.643625 4303 sys_catalog.cc:339] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "8a3f32f20ddc40d986bae579a55da0f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ccf046f8f4748bc9dea8828a7a48863" member_type: VOTER last_known_addr { host: "127.2.5.62" port: 35093 } } peers { permanent_uuid: "513938f6acf044ed88d2d88de9565cad" member_type: VOTER last_known_addr { host: "127.2.5.61" port: 35161 } } peers { permanent_uuid: "8a3f32f20ddc40d986bae579a55da0f1" member_type: VOTER last_known_addr { host: "127.2.5.60" port: 39705 } } } 04:10:34.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.644490 4303 sys_catalog.cc:342] T 00000000000000000000000000000000 P 8a3f32f20ddc40d986bae579a55da0f1 [sys.catalog]: This master's current role is: LEADER 04:10:34.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.645802 4305 sys_catalog.cc:342] T 00000000000000000000000000000000 P 7ccf046f8f4748bc9dea8828a7a48863 [sys.catalog]: This master's current role is: FOLLOWER 04:10:34.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.655349 4253 rpc_server.cc:205] RPC server started. Bound to: 127.2.5.61:35161 04:10:34.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.658396 4253 webserver.cc:374] Webserver started at http://127.2.5.61:45345/ using document root and password file 04:10:34.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.659344 4253 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/master-1/data/info.pb 04:10:34.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.664378 2068 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4253 04:10:34.664 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:34.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.667871 4335 sys_catalog.cc:188] Verifying existing consensus state 04:10:34.669 [DEBUG - New I/O worker #57] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:10:34.670 [DEBUG - New I/O worker #57] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster363434838761724841/krb5kdc/krb5cc 04:10:34.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.672328 4335 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap starting. 04:10:34.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:34 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:34.674 [DEBUG - New I/O worker #57] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:10:34.674 [INFO - New I/O worker #57] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:10:34.675 [DEBUG - New I/O worker #57] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.2.5.60 04:10:34.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:34 dist-test-slave-dist-test-slave-fvsv krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.5.60@KRBTEST.COM 04:10:34.704 [DEBUG - New I/O worker #57] (Negotiator.java:764) Authenticated connection [id: 0x78af3866, /127.0.0.1:58502 => /127.2.5.60:39705] using SASL/GSSAPI 04:10:34.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.710198 4335 log.cc:525] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Log is configured to *not* fsync() on all Append() calls 04:10:34.711 [DEBUG - New I/O worker #57] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:10:34.712 [DEBUG - New I/O worker #57] (Connection.java:691) [peer master-127.2.5.60:39705(127.2.5.60:39705)] cleaning up while in state READY due to: connection disconnected 04:10:34.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.712894 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4186 04:10:34.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.728075 4335 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:34.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.729045 4335 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 513938f6acf044ed88d2d88de9565cad: Bootstrap complete. 04:10:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.747644 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4253 04:10:34.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:34.762034 2068 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4123 04:10:35.809 [INFO - main] (RetryRule.java:136) Creating RetryStatement with result reporter and retry count of 0 (no retries) .04:10:35.810 [INFO - Time-limited test] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskusvpEt/test-tmp 04:10:35.810 [INFO - Time-limited test] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin 04:10:35.811 [INFO - Time-limited test] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskusvpEt/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb] 04:10:35.812 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 0 enable_kerberos: true cluster_root: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663" extra_master_flags: "--authn_token_validity_seconds=10" hms_mode: NONE mini_kdc_options { ticket_lifetime: "10s" renew_lifetime: "20s" } } 04:10:35.985 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:35.985 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_cluster { } 04:10:36.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Loading random data 04:10:36.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Initializing database '/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/principal' for realm 'KRBTEST.COM', 04:10:36.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) master key name 'K/M@KRBTEST.COM' 04:10:36.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): setting up network... 04:10:36.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) 04:10:36.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked 04:10:36.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 04:10:36.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): skipping unrecognized local address family 17 04:10:36.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): skipping unrecognized local address family 17 04:10:36.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): set up 1 sockets 04:10:36.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): commencing operation 04:10:36.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: starting... 04:10:36.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:36.567626 4341 mini_kdc.cc:119] Time spent starting KDC: real 0.555s user 0.003s sys 0.014s 04:10:36.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy 04:10:36.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:10:36.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-admin@KRBTEST.COM" created. 04:10:36.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy 04:10:36.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:10:36.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-user@KRBTEST.COM" created. 04:10:36.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy 04:10:36.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:10:36.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "joe-interloper@KRBTEST.COM" created. 04:10:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:10:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/test-user.keytab. 04:10:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/test-user.keytab. 04:10:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/test-user.keytab. 04:10:36.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/test-user.keytab. 04:10:36.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:36.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:10:36.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.4.61.126@KRBTEST.COM; defaulting to no policy 04:10:36.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:10:36.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.4.61.126@KRBTEST.COM" created. 04:10:36.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:10:36.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.4.61.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.126.keytab. 04:10:36.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.4.61.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.126.keytab. 04:10:36.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.4.61.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.126.keytab. 04:10:36.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.4.61.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.126.keytab. 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.720822 4341 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.126:46743 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.126 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:36.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/logs 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/info.pb 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.126.keytab 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.126 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:36.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:36.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:36.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:36.864591 4361 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:36.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:36.864918 4361 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.4.61.126 04:10:36.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:36.865162 4361 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:36.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:36.876983 4361 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:36.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:36.877121 4361 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:36.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:36.877225 4361 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:36.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:36.877315 4361 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:36.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:36.877434 4361 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:36.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:36.877552 4361 flags.cc:404] Enabled experimental flag: --principal=kudu/127.4.61.126 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.895793 4361 master_main.cc:98] Master server non-default flags: 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.126:46743 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.126.keytab 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.126 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/info.pb 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.126 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:36.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:36.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/logs 04:10:36.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:36.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:36.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:36.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:36.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:36.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:36.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:36.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:36.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:36.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.909720 4361 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 155016us 04:10:36.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608636, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:36.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.911994 4361 init.cc:359] Logged in from keytab as kudu/127.4.61.126@KRBTEST.COM (short username kudu) 04:10:36.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.914666 4361 fs_manager.cc:263] Metadata directory not provided 04:10:36.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.914803 4361 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:36.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.915144 4361 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:10:36.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.915262 4361 server_base.cc:449] Attempting to create new FS layout instead 04:10:36.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.918171 4361 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/instance: 04:10:36.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "ac1846f7dc19401ebfba50d7f54975fe" 04:10:36.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:36 on dist-test-slave-dist-test-slave-fvsv" 04:10:36.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.918952 4361 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal/instance: 04:10:36.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "ac1846f7dc19401ebfba50d7f54975fe" 04:10:36.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:36 on dist-test-slave-dist-test-slave-fvsv" 04:10:36.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.927605 4361 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s 04:10:36.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.928787 4361 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:36.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.929311 4361 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:36.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.935894 4361 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s 04:10:36.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.936022 4361 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:36.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "ac1846f7dc19401ebfba50d7f54975fe" 04:10:36.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:36 on dist-test-slave-dist-test-slave-fvsv" 04:10:36.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:36.936295 4361 fs_report.cc:352] FS layout report 04:10:36.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:36.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:36.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:36.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/data 04:10:36.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:36.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:36.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:36.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:36.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:36.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:36.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:36.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:36.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:36.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:37.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.581521 4361 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.614s user 0.001s sys 0.000s 04:10:37.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.581714 4361 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.615s user 0.001s sys 0.000s 04:10:37.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.627391 4361 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:37.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.701246 4361 rpc_server.cc:205] RPC server started. Bound to: 127.4.61.126:46743 04:10:37.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.704044 4361 webserver.cc:374] Webserver started at http://127.4.61.126:35121/ using document root and password file 04:10:37.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.704905 4361 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/info.pb 04:10:37.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.708034 4423 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:10:37.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.712570 4341 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4361 04:10:37.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.721676 4423 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } has no permanent_uuid. Determining permanent_uuid... 04:10:37.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.4.61.125@KRBTEST.COM; defaulting to no policy 04:10:37.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:10:37.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.4.61.125@KRBTEST.COM" created. 04:10:37.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:37 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608636, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.126@KRBTEST.COM for kudu/127.4.61.126@KRBTEST.COM 04:10:37.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:10:37.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.4.61.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.125.keytab. 04:10:37.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.4.61.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.125.keytab. 04:10:37.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.4.61.125 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.125.keytab. 04:10:37.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.4.61.125 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.125.keytab. 04:10:37.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.752573 4341 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:37.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:37.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:37.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data 04:10:37.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.125:46275 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.125 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/logs 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/info.pb 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.125.keytab 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.125 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:37.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:37.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:37.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:37.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.752852 4423 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } has no permanent_uuid. Determining permanent_uuid... 04:10:37.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.754686 4423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.125:46275: Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111) 04:10:37.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.774101 4423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } attempt: 1 04:10:37.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.775609 4423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.125:46275: Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111) 04:10:37.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.823987 4423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } attempt: 2 04:10:37.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.825662 4423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.125:46275: Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111) 04:10:37.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:37.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.899261 4428 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:37.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.899581 4428 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.4.61.125 04:10:37.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.899811 4428 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:37.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.910663 4428 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:37.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.910809 4428 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:37.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.910912 4428 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.911003 4428 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.911180 4428 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:37.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.911305 4428 flags.cc:404] Enabled experimental flag: --principal=kudu/127.4.61.125 04:10:37.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.930480 4428 master_main.cc:98] Master server non-default flags: 04:10:37.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data 04:10:37.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:37.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:37.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:37.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.125:46275 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.125.keytab 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.125 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/info.pb 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.125 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/logs 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:37.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:37.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.937006 4423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } attempt: 3 04:10:37.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:37.938570 4423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.125:46275: Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111) 04:10:37.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.944712 4428 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 155516us 04:10:37.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:37 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608637, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:37.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.946903 4428 init.cc:359] Logged in from keytab as kudu/127.4.61.125@KRBTEST.COM (short username kudu) 04:10:37.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.950667 4428 fs_manager.cc:263] Metadata directory not provided 04:10:37.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.950789 4428 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:37.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.951015 4428 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:10:37.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.951169 4428 server_base.cc:449] Attempting to create new FS layout instead 04:10:37.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.954042 4428 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/instance: 04:10:37.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1358655cf490440fbff9f81ab4ea9168" 04:10:37.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:37 on dist-test-slave-dist-test-slave-fvsv" 04:10:37.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.954735 4428 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal/instance: 04:10:37.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1358655cf490440fbff9f81ab4ea9168" 04:10:37.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:37 on dist-test-slave-dist-test-slave-fvsv" 04:10:37.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.963431 4428 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.007s sys 0.004s 04:10:37.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.964594 4428 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:37.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.965091 4428 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:37.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.971568 4428 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:10:37.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.971685 4428 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:37.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1358655cf490440fbff9f81ab4ea9168" 04:10:37.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:37 on dist-test-slave-dist-test-slave-fvsv" 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:37.971899 4428 fs_report.cc:352] FS layout report 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/data 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:37.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:38.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.114936 4423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } attempt: 4 04:10:38.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.117324 4423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.125:46275: Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111) 04:10:38.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.420720 4423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } attempt: 5 04:10:38.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.423032 4423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.125:46275: Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111) 04:10:38.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.602162 4428 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.606s user 0.282s sys 0.324s 04:10:38.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.602443 4428 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.607s user 0.282s sys 0.324s 04:10:38.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.643541 4428 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:38.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.717123 4428 rpc_server.cc:205] RPC server started. Bound to: 127.4.61.125:46275 04:10:38.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.719915 4428 webserver.cc:374] Webserver started at http://127.4.61.125:35773/ using document root and password file 04:10:38.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.720754 4428 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/info.pb 04:10:38.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.724365 4490 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:10:38.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.726586 4341 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4428 04:10:38.738 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.4.61.124@KRBTEST.COM; defaulting to no policy 04:10:38.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:10:38.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.4.61.124@KRBTEST.COM" created. 04:10:38.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.745378 4490 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } has no permanent_uuid. Determining permanent_uuid... 04:10:38.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:38 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608637, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.125@KRBTEST.COM for kudu/127.4.61.126@KRBTEST.COM 04:10:38.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:10:38.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.4.61.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.124.keytab. 04:10:38.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.4.61.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.124.keytab. 04:10:38.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.4.61.124 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.124.keytab. 04:10:38.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.4.61.124 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.124.keytab. 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.766819 4341 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.124:36431 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.124 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/logs 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/info.pb 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.124.keytab 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.124 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:38.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:38.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:38.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.775454 4490 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } has no permanent_uuid. Determining permanent_uuid... 04:10:38.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:38 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608637, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.125@KRBTEST.COM for kudu/127.4.61.125@KRBTEST.COM 04:10:38.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.796314 4490 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } has no permanent_uuid. Determining permanent_uuid... 04:10:38.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.797751 4490 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.124:36431: Network error: Client connection negotiation failed: client connection to 127.4.61.124:36431: connect: Connection refused (error 111) 04:10:38.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.817080 4490 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } attempt: 1 04:10:38.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.818591 4490 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.124:36431: Network error: Client connection negotiation failed: client connection to 127.4.61.124:36431: connect: Connection refused (error 111) 04:10:38.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.866943 4490 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } attempt: 2 04:10:38.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.868705 4490 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.124:36431: Network error: Client connection negotiation failed: client connection to 127.4.61.124:36431: connect: Connection refused (error 111) 04:10:38.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:38.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.919668 4495 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:38.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.919998 4495 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.4.61.124 04:10:38.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.920231 4495 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:38.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.931150 4495 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:38.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.931288 4495 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:38.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.931393 4495 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:38.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.931483 4495 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:38.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.931596 4495 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:38.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.931720 4495 flags.cc:404] Enabled experimental flag: --principal=kudu/127.4.61.124 04:10:38.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.949525 4495 master_main.cc:98] Master server non-default flags: 04:10:38.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data 04:10:38.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:38.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:38.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.124:36431 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.124.keytab 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.124 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/info.pb 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.124 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/logs 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:38.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:38.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.957404 4423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } attempt: 6 04:10:38.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.963982 4495 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 156016us 04:10:38.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:38 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608638, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:38.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:38 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608636, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.126@KRBTEST.COM for kudu/127.4.61.125@KRBTEST.COM 04:10:38.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.966245 4495 init.cc:359] Logged in from keytab as kudu/127.4.61.124@KRBTEST.COM (short username kudu) 04:10:38.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.969748 4495 fs_manager.cc:263] Metadata directory not provided 04:10:38.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.969887 4495 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:38.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.970216 4495 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:10:38.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.970337 4495 server_base.cc:449] Attempting to create new FS layout instead 04:10:38.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.973335 4495 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/instance: 04:10:38.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "9a549346b3a942b28912139226b10196" 04:10:38.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:38 on dist-test-slave-dist-test-slave-fvsv" 04:10:38.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.974207 4495 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal/instance: 04:10:38.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "9a549346b3a942b28912139226b10196" 04:10:38.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:38 on dist-test-slave-dist-test-slave-fvsv" 04:10:38.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.975188 4423 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } has no permanent_uuid. Determining permanent_uuid... 04:10:38.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.976480 4423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.124:36431: Network error: Client connection negotiation failed: client connection to 127.4.61.124:36431: connect: Connection refused (error 111) 04:10:38.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.980015 4490 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } attempt: 3 04:10:38.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:38.981441 4490 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.124:36431: Network error: Client connection negotiation failed: client connection to 127.4.61.124:36431: connect: Connection refused (error 111) 04:10:38.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.983673 4495 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.011s sys 0.001s 04:10:38.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.984917 4495 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:38.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.985419 4495 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:38.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.992664 4495 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s 04:10:38.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.992786 4495 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "9a549346b3a942b28912139226b10196" 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:38 on dist-test-slave-dist-test-slave-fvsv" 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:38.992998 4495 fs_report.cc:352] FS layout report 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/data 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:38.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:39.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.008774 4423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } attempt: 1 04:10:39.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:39.010270 4423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.124:36431: Network error: Client connection negotiation failed: client connection to 127.4.61.124:36431: connect: Connection refused (error 111) 04:10:39.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.078588 4423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } attempt: 2 04:10:39.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:39.080318 4423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.124:36431: Network error: Client connection negotiation failed: client connection to 127.4.61.124:36431: connect: Connection refused (error 111) 04:10:39.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.156605 4423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } attempt: 3 04:10:39.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:39.158288 4423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.124:36431: Network error: Client connection negotiation failed: client connection to 127.4.61.124:36431: connect: Connection refused (error 111) 04:10:39.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.158164 4490 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } attempt: 4 04:10:39.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:39.159700 4490 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.124:36431: Network error: Client connection negotiation failed: client connection to 127.4.61.124:36431: connect: Connection refused (error 111) 04:10:39.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.295578 4423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } attempt: 4 04:10:39.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:39.297178 4423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.124:36431: Network error: Client connection negotiation failed: client connection to 127.4.61.124:36431: connect: Connection refused (error 111) 04:10:39.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.462967 4490 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } attempt: 5 04:10:39.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:39.464629 4490 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.124:36431: Network error: Client connection negotiation failed: client connection to 127.4.61.124:36431: connect: Connection refused (error 111) 04:10:39.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.580457 4423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } attempt: 5 04:10:39.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:39.582175 4423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.4.61.124:36431: Network error: Client connection negotiation failed: client connection to 127.4.61.124:36431: connect: Connection refused (error 111) 04:10:39.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:39.862602 4495 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.846s user 0.324s sys 0.521s 04:10:39.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:39.862808 4495 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.846s user 0.324s sys 0.521s 04:10:39.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.913044 4495 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:39.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.980566 4495 rpc_server.cc:205] RPC server started. Bound to: 127.4.61.124:36431 04:10:39.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.982872 4495 webserver.cc:374] Webserver started at http://127.4.61.124:38787/ using document root and password file 04:10:39.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.983819 4495 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/info.pb 04:10:39.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.987656 4559 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:10:39.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.988607 4341 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4495 04:10:39.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:39.999020 4490 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } attempt: 6 04:10:40.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.010628 4559 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } has no permanent_uuid. Determining permanent_uuid... 04:10:40.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:40 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608637, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.125@KRBTEST.COM for kudu/127.4.61.124@KRBTEST.COM 04:10:40.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:40 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.61.124@KRBTEST.COM 04:10:40.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:40 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608638, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.124@KRBTEST.COM for kudu/127.4.61.126@KRBTEST.COM 04:10:40.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.037695 4559 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } has no permanent_uuid. Determining permanent_uuid... 04:10:40.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.041458 4490 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Bootstrap starting. 04:10:40.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.047631 4490 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: No blocks or log segments found. Creating new log. 04:10:40.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:40 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608638, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.124@KRBTEST.COM for kudu/127.4.61.125@KRBTEST.COM 04:10:40.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.049651 4490 log.cc:525] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Log is configured to *not* fsync() on all Append() calls 04:10:40.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:40 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.61.125@KRBTEST.COM 04:10:40.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.053293 4490 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: No bootstrap required, opened a new log 04:10:40.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.065469 4559 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } has no permanent_uuid. Determining permanent_uuid... 04:10:40.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.069357 4490 raft_consensus.cc:340] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:40.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.069949 4490 raft_consensus.cc:366] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:10:40.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.070158 4490 raft_consensus.cc:705] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1358655cf490440fbff9f81ab4ea9168, State: Initialized, Role: FOLLOWER 04:10:40.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.070819 4490 consensus_queue.cc:229] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:40.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.073395 4490 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:40 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.61.126@KRBTEST.COM 04:10:40.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.074035 4490 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: FOLLOWER 04:10:40.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.075146 4490 sys_catalog.cc:427] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: configured and running, proceeding with master startup. 04:10:40.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:40 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608638, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.124@KRBTEST.COM for kudu/127.4.61.124@KRBTEST.COM 04:10:40.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.094590 4423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } attempt: 6 04:10:40.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.095984 4341 external_mini_cluster.cc:628] 0 TS(s) registered with all masters 04:10:40.096 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:40.096 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_kdc_env_vars { } 04:10:40.097 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: get_kdc_env_vars { env_vars { key: "KRB5CCNAME" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc" } env_vars { key: "KUDU_ENABLE_KRB5_REALM_FIX" value: "yes" } env_vars { key: "KRB5_KDC_PROFILE" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf" } env_vars { key: "KRB5_CONFIG" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf" } } 04:10:40.098 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_masters { } 04:10:40.099 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.4.61.126" port: 46743 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.4.61.125" port: 46275 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.4.61.124" port: 36431 } } } 04:10:40.099 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_tservers { } 04:10:40.099 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:40.099 [INFO - Time-limited test] (MiniKuduCluster.java:439) Running kinit for user test-admin 04:10:40.099 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: kinit { username: "test-admin" } 04:10:40.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.103421 4559 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Bootstrap starting. 04:10:40.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.112303 4559 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: No blocks or log segments found. Creating new log. 04:10:40.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.116179 4559 log.cc:525] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Log is configured to *not* fsync() on all Append() calls 04:10:40.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:40 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608636, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.126@KRBTEST.COM for kudu/127.4.61.124@KRBTEST.COM 04:10:40.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.134131 4559 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: No bootstrap required, opened a new log 04:10:40.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:40 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:40.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:10:40.146 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:40.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.146255 4423 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Bootstrap starting. 04:10:40.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.149940 4423 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: No blocks or log segments found. Creating new log. 04:10:40.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.152007 4423 log.cc:525] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Log is configured to *not* fsync() on all Append() calls 04:10:40.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.161072 4423 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: No bootstrap required, opened a new log 04:10:40.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.164777 4559 raft_consensus.cc:340] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:40.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.165519 4559 raft_consensus.cc:366] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:10:40.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.165814 4559 raft_consensus.cc:705] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a549346b3a942b28912139226b10196, State: Initialized, Role: FOLLOWER 04:10:40.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.166611 4559 consensus_queue.cc:229] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:40.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.169148 4559 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.169834 4559 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:10:40.170 [DEBUG - Time-limited test] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc 04:10:40.171 [DEBUG - Time-limited test] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:10:40.172 [INFO - Time-limited test] (MiniKuduCluster.java:439) Running kinit for user test-user 04:10:40.172 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: kinit { username: "test-user" } 04:10:40.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.171152 4559 sys_catalog.cc:427] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: configured and running, proceeding with master startup. 04:10:40.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.182909 4423 raft_consensus.cc:340] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:40.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.183522 4423 raft_consensus.cc:366] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:10:40.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.183728 4423 raft_consensus.cc:705] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1846f7dc19401ebfba50d7f54975fe, State: Initialized, Role: FOLLOWER 04:10:40.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.184324 4423 consensus_queue.cc:229] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:40.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.073644 4569 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.200953 4569 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: FOLLOWER 04:10:40.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.217012 4423 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.217797 4423 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: FOLLOWER 04:10:40.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.219683 4423 sys_catalog.cc:427] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: configured and running, proceeding with master startup. 04:10:40.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:40 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:40.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-user@KRBTEST.COM: 04:10:40.259 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:40.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.261245 4569 raft_consensus.cc:472] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:10:40.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.261673 4569 raft_consensus.cc:494] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:40.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.263823 4569 leader_election.cc:296] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac1846f7dc19401ebfba50d7f54975fe (127.4.61.126:46743), 9a549346b3a942b28912139226b10196 (127.4.61.124:36431) 04:10:40.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:40.150312 4446 tablet.cc:1799] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:10:40.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.267556 4570 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:10:40.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.169409 4585 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.286916 4585 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:10:40.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.219413 4588 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.324786 4588 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: FOLLOWER 04:10:40.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.266296 4536 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1358655cf490440fbff9f81ab4ea9168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a549346b3a942b28912139226b10196" is_pre_election: true 04:10:40.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.336714 4536 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1358655cf490440fbff9f81ab4ea9168 in term 0. 04:10:40.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.338335 4437 leader_election.cc:310] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1358655cf490440fbff9f81ab4ea9168, 9a549346b3a942b28912139226b10196; no voters: 04:10:40.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.339412 4569 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 0 FOLLOWER]: Leader pre-election won for term 1 04:10:40.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.339684 4569 raft_consensus.cc:472] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:10:40.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.339916 4569 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 0 FOLLOWER]: Advancing to term 1 04:10:40.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:40.344547 4586 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:10:40.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:40.344792 4586 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:10:40.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.344969 4569 raft_consensus.cc:494] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:40.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.347468 4399 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1358655cf490440fbff9f81ab4ea9168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1846f7dc19401ebfba50d7f54975fe" 04:10:40.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.348232 4399 raft_consensus.cc:2890] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 0 FOLLOWER]: Advancing to term 1 04:10:40.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.353205 4399 raft_consensus.cc:2350] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1358655cf490440fbff9f81ab4ea9168 in term 1. 04:10:40.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.354306 4437 leader_election.cc:310] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1358655cf490440fbff9f81ab4ea9168, ac1846f7dc19401ebfba50d7f54975fe; no voters: 04:10:40.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.356572 4569 leader_election.cc:296] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [CANDIDATE]: Term 1 election: Requested vote from peers ac1846f7dc19401ebfba50d7f54975fe (127.4.61.126:46743), 9a549346b3a942b28912139226b10196 (127.4.61.124:36431) 04:10:40.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.356953 4569 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 1 FOLLOWER]: Leader election won for term 1 04:10:40.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.357278 4569 raft_consensus.cc:667] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 1 LEADER]: Becoming Leader. State: Replica: 1358655cf490440fbff9f81ab4ea9168, State: Running, Role: LEADER 04:10:40.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.357816 4569 consensus_queue.cc:206] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:40.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.360690 4536 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1358655cf490440fbff9f81ab4ea9168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a549346b3a942b28912139226b10196" 04:10:40.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.361218 4536 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 0 FOLLOWER]: Advancing to term 1 04:10:40.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.366889 4536 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1358655cf490440fbff9f81ab4ea9168 in term 1. 04:10:40.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.380084 4592 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1358655cf490440fbff9f81ab4ea9168. Latest consensus state: current_term: 1 leader_uuid: "1358655cf490440fbff9f81ab4ea9168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.380906 4592 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: LEADER 04:10:40.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.267161 4400 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1358655cf490440fbff9f81ab4ea9168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1846f7dc19401ebfba50d7f54975fe" is_pre_election: true 04:10:40.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:40.384310 4379 tablet.cc:1799] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:10:40.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.384260 4400 raft_consensus.cc:2263] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1358655cf490440fbff9f81ab4ea9168 in term 1. Re-sending same reply. 04:10:40.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.388260 4594 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:10:40.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.398331 4594 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:10:40.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:40.400148 4590 catalog_manager.cc:1122] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:10:40.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:40.400524 4590 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:10:40.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:40.418942 4515 tablet.cc:1799] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:10:40.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.465823 4536 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 1 FOLLOWER]: Refusing update from remote peer 1358655cf490440fbff9f81ab4ea9168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:10:40.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.467890 4569 consensus_queue.cc:984] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:10:40.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.469666 4400 raft_consensus.cc:1184] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 1 FOLLOWER]: Refusing update from remote peer 1358655cf490440fbff9f81ab4ea9168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:10:40.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.471107 4569 consensus_queue.cc:984] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:10:40.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.486277 4585 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1358655cf490440fbff9f81ab4ea9168. Latest consensus state: current_term: 1 leader_uuid: "1358655cf490440fbff9f81ab4ea9168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.486997 4585 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:10:40.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.490298 4592 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1358655cf490440fbff9f81ab4ea9168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.491117 4592 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: LEADER 04:10:40.493 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.492980 4598 mvcc.cc:203] Tried to move safe_time back from 6396348991281786880 to 6396348990933262336. Current Snapshot: MvccSnapshot[committed={T|T < 6396348991281786880}] 04:10:40.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.493872 4594 catalog_manager.cc:959] Generated new certificate authority record 04:10:40.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.494446 4588 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1358655cf490440fbff9f81ab4ea9168. Latest consensus state: current_term: 1 leader_uuid: "1358655cf490440fbff9f81ab4ea9168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.495285 4588 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: FOLLOWER 04:10:40.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.496306 4594 catalog_manager.cc:1084] Loading token signing keys... 04:10:40.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.501471 4592 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1358655cf490440fbff9f81ab4ea9168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.502095 4592 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: LEADER 04:10:40.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.508719 4585 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1358655cf490440fbff9f81ab4ea9168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.509383 4585 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:10:40.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.510555 4588 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1358655cf490440fbff9f81ab4ea9168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:40.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.511188 4588 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: FOLLOWER 04:10:40.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.513368 4600 mvcc.cc:203] Tried to move safe_time back from 6396348991281786880 to 6396348990933262336. Current Snapshot: MvccSnapshot[committed={T|T < 6396348991281786880 or (T in {6396348991281786880})}] 04:10:40.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:40.522907 4594 catalog_manager.cc:4350] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Generated new TSK 0 04:10:41.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:41.351342 4586 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: acquiring CA information for follower catalog manager: success 04:10:41.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:41.354683 4586 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:41.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:41.408046 4590 catalog_manager.cc:1120] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: acquiring CA information for follower catalog manager: success 04:10:41.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:41.411324 4590 catalog_manager.cc:1148] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:43.262 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.4.61.126:46743 04:10:43.263 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:10:43.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.263679 4341 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4361 04:10:43.282 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:43.282 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.4.61.124:36431 04:10:43.283 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:10:43.283 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.283578 4341 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4495 04:10:43.301 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:43.302 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.4.61.125:46275 04:10:43.302 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:10:43.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.302616 4341 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4428 04:10:43.326 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:43.326 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.4.61.126:46743 04:10:43.326 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:10:43.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.327507 4341 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:43.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:43.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:43.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data 04:10:43.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:43.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.126:46743 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.126 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=35121 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/logs 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/info.pb 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.126.keytab 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.126 04:10:43.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:43.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:43.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:43.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:43.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:43.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:43.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:43.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:43.473843 4605 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:43.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:43.474169 4605 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.4.61.126 04:10:43.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:43.474400 4605 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:43.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:43.485154 4605 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:43.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:43.485291 4605 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:43.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:43.485404 4605 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:43.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:43.485496 4605 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:43.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:43.485637 4605 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:43.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:43.485762 4605 flags.cc:404] Enabled experimental flag: --principal=kudu/127.4.61.126 04:10:43.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.503527 4605 master_main.cc:98] Master server non-default flags: 04:10:43.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data 04:10:43.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:43.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:43.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.126:46743 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.126.keytab 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.126 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/info.pb 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.126 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=35121 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/logs 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:43.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.517863 4605 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 158516us 04:10:43.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:43 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608643, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:43.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.520184 4605 init.cc:359] Logged in from keytab as kudu/127.4.61.126@KRBTEST.COM (short username kudu) 04:10:43.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.522902 4605 fs_manager.cc:263] Metadata directory not provided 04:10:43.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.523036 4605 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:43.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.527994 4605 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s 04:10:43.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.528635 4605 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:43.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.529124 4605 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:43.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.536195 4605 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s 04:10:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.536320 4605 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "ac1846f7dc19401ebfba50d7f54975fe" 04:10:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:36 on dist-test-slave-dist-test-slave-fvsv" 04:10:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:43.537554 4605 fs_report.cc:352] FS layout report 04:10:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/data 04:10:43.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:43.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:44.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:44.197491 4605 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.634s user 0.000s sys 0.002s 04:10:44.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:44.197691 4605 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.634s user 0.000s sys 0.002s 04:10:44.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.224946 4605 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:44.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.313371 4605 rpc_server.cc:205] RPC server started. Bound to: 127.4.61.126:46743 04:10:44.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.315699 4605 webserver.cc:374] Webserver started at http://127.4.61.126:35121/ using document root and password file 04:10:44.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.316617 4605 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/info.pb 04:10:44.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.320400 4341 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4605 04:10:44.320 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:44.321 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.4.61.124:36431 04:10:44.321 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:10:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.322211 4341 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data 04:10:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.124:36431 04:10:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.124 04:10:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=38787 04:10:44.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/logs 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/info.pb 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.124.keytab 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.124 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:44.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:44.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:44.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:44.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.324251 4667 sys_catalog.cc:188] Verifying existing consensus state 04:10:44.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.332341 4667 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Bootstrap starting. 04:10:44.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.389199 4667 log.cc:525] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Log is configured to *not* fsync() on all Append() calls 04:10:44.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.404333 4667 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:44.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.405145 4667 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Bootstrap complete. 04:10:44.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.435883 4667 raft_consensus.cc:340] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:44.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.443905 4667 raft_consensus.cc:705] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1846f7dc19401ebfba50d7f54975fe, State: Initialized, Role: FOLLOWER 04:10:44.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.444730 4667 consensus_queue.cc:229] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:44.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.447911 4667 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:44.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.448590 4667 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: FOLLOWER 04:10:44.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.449697 4667 sys_catalog.cc:427] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: configured and running, proceeding with master startup. 04:10:44.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.448240 4670 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:44.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.547168 4670 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: FOLLOWER 04:10:44.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:44.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:44.552690 4668 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:44.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:44.553104 4668 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.4.61.124 04:10:44.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:44.553472 4668 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:44.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.555202 4671 catalog_manager.cc:1120] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: acquiring CA information for follower catalog manager: success 04:10:44.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.557137 4671 catalog_manager.cc:1148] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:44.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:44.572471 4668 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:44.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:44.572681 4668 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:44.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:44.572865 4668 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:44.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:44.573022 4668 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:44.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:44.573209 4668 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:44.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:44.573410 4668 flags.cc:404] Enabled experimental flag: --principal=kudu/127.4.61.124 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.612980 4668 master_main.cc:98] Master server non-default flags: 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.124:36431 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.124.keytab 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.124 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:44.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/info.pb 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.124 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=38787 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/logs 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:44.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:44.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.636742 4668 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 159016us 04:10:44.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:44 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608644, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:44.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.639578 4668 init.cc:359] Logged in from keytab as kudu/127.4.61.124@KRBTEST.COM (short username kudu) 04:10:44.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.645162 4668 fs_manager.cc:263] Metadata directory not provided 04:10:44.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.645368 4668 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:44.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.652861 4668 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s 04:10:44.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.653776 4668 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:44.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.654479 4668 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:44.668 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.668697 4668 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s 04:10:44.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.668885 4668 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:44.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "9a549346b3a942b28912139226b10196" 04:10:44.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:38 on dist-test-slave-dist-test-slave-fvsv" 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:44.670784 4668 fs_report.cc:352] FS layout report 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/data 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:44.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:45.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:45.579139 4668 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.874s user 0.369s sys 0.504s 04:10:45.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:45.579340 4668 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.874s user 0.369s sys 0.505s 04:10:45.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.640224 4668 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:45.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.755836 4668 rpc_server.cc:205] RPC server started. Bound to: 127.4.61.124:36431 04:10:45.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.759816 4668 webserver.cc:374] Webserver started at http://127.4.61.124:38787/ using document root and password file 04:10:45.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.761011 4668 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/info.pb 04:10:45.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.770471 4341 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4668 04:10:45.770 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:45.771 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.4.61.125:46275 04:10:45.771 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.772224 4341 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.125:46275 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.125 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=35773 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:45.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/logs 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/info.pb 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.125.keytab 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.125 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:45.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.773501 4734 sys_catalog.cc:188] Verifying existing consensus state 04:10:45.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.785478 4734 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Bootstrap starting. 04:10:45.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.868923 4734 log.cc:525] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Log is configured to *not* fsync() on all Append() calls 04:10:45.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.888350 4734 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:45.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.889199 4734 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Bootstrap complete. 04:10:45.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.919674 4734 raft_consensus.cc:340] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:45.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.927270 4734 raft_consensus.cc:705] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a549346b3a942b28912139226b10196, State: Initialized, Role: FOLLOWER 04:10:45.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.928012 4734 consensus_queue.cc:229] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:45.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.931931 4734 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:45.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.932564 4734 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:10:45.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.933660 4734 sys_catalog.cc:427] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: configured and running, proceeding with master startup. 04:10:45.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:45.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:45.993880 4735 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:45.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:45.994238 4735 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.4.61.125 04:10:45.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:45.994555 4735 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:46.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:46.007084 4735 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:46.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:46.007237 4735 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:46.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:46.007383 4735 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:46.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:46.007514 4735 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:46.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:46.007699 4735 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:46.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:46.007869 4735 flags.cc:404] Enabled experimental flag: --principal=kudu/127.4.61.125 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.027920 4735 master_main.cc:98] Master server non-default flags: 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.125:46275 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.125.keytab 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.125 04:10:46.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/info.pb 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.125 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=35773 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/logs 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:46.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:46.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:46.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:46.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:45.932341 4737 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:46.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.041672 4737 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:10:46.046 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.045917 4735 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 160016us 04:10:46.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:46 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608646, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:46.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.048404 4735 init.cc:359] Logged in from keytab as kudu/127.4.61.125@KRBTEST.COM (short username kudu) 04:10:46.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.052603 4738 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: acquiring CA information for follower catalog manager: success 04:10:46.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.054482 4738 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:46.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.059803 4735 fs_manager.cc:263] Metadata directory not provided 04:10:46.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.059972 4735 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:46.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.065433 4735 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s 04:10:46.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.066109 4735 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:46.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.066650 4735 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:46.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.073940 4735 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:10:46.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.074074 4735 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:46.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1358655cf490440fbff9f81ab4ea9168" 04:10:46.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:37 on dist-test-slave-dist-test-slave-fvsv" 04:10:46.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.075435 4735 fs_report.cc:352] FS layout report 04:10:46.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:46.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:46.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:46.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/data 04:10:46.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:46.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:46.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:46.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:46.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:46.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:46.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:46.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:46.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:46.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:46.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.271891 4753 raft_consensus.cc:472] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:10:46.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.272294 4753 raft_consensus.cc:494] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:46.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.277011 4753 leader_election.cc:296] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275), 9a549346b3a942b28912139226b10196 (127.4.61.124:36431) 04:10:46.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:46.277547 4618 leader_election.cc:341] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275): Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111) 04:10:46.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.297266 4711 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac1846f7dc19401ebfba50d7f54975fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9a549346b3a942b28912139226b10196" is_pre_election: true 04:10:46.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.298087 4711 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac1846f7dc19401ebfba50d7f54975fe in term 1. 04:10:46.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.299715 4614 leader_election.cc:310] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a549346b3a942b28912139226b10196, ac1846f7dc19401ebfba50d7f54975fe; no voters: 1358655cf490440fbff9f81ab4ea9168 04:10:46.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.300827 4753 raft_consensus.cc:2644] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 1 FOLLOWER]: Leader pre-election won for term 2 04:10:46.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.301065 4753 raft_consensus.cc:472] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:10:46.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.301278 4753 raft_consensus.cc:2890] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 1 FOLLOWER]: Advancing to term 2 04:10:46.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.308066 4753 raft_consensus.cc:494] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:46.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:46.310668 4618 leader_election.cc:341] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275): Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111) 04:10:46.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.312239 4711 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac1846f7dc19401ebfba50d7f54975fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9a549346b3a942b28912139226b10196" 04:10:46.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.312711 4711 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 1 FOLLOWER]: Advancing to term 2 04:10:46.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.318740 4711 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac1846f7dc19401ebfba50d7f54975fe in term 2. 04:10:46.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.319394 4753 leader_election.cc:296] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 2 election: Requested vote from peers 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275), 9a549346b3a942b28912139226b10196 (127.4.61.124:36431) 04:10:46.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.320477 4614 leader_election.cc:310] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a549346b3a942b28912139226b10196, ac1846f7dc19401ebfba50d7f54975fe; no voters: 1358655cf490440fbff9f81ab4ea9168 04:10:46.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.321473 4753 raft_consensus.cc:2644] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 2 FOLLOWER]: Leader election won for term 2 04:10:46.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.325765 4753 raft_consensus.cc:667] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 2 LEADER]: Becoming Leader. State: Replica: ac1846f7dc19401ebfba50d7f54975fe, State: Running, Role: LEADER 04:10:46.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.326488 4753 consensus_queue.cc:206] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:46.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.333986 4757 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1846f7dc19401ebfba50d7f54975fe. Latest consensus state: current_term: 2 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:46.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.334905 4757 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: LEADER 04:10:46.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.337178 4759 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:10:46.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.341074 4759 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:10:46.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.343979 4759 catalog_manager.cc:1084] Loading token signing keys... 04:10:46.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.345937 4759 catalog_manager.cc:4361] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Loaded TSK: 0 04:10:46.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.878123 4711 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 2 FOLLOWER]: Refusing update from remote peer ac1846f7dc19401ebfba50d7f54975fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) 04:10:46.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.880512 4760 consensus_queue.cc:984] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s 04:10:46.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.892720 4761 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1846f7dc19401ebfba50d7f54975fe. Latest consensus state: current_term: 2 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:46.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.895352 4761 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:10:46.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.905082 4761 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:46.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.905894 4761 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:10:46.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.915827 4760 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:46.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:46.916507 4618 consensus_peers.cc:458] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe -> Peer 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275): Couldn't send request to peer 1358655cf490440fbff9f81ab4ea9168. Status: Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:10:46.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:46.917130 4760 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: LEADER 04:10:46.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:46.991168 4735 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.884s user 0.000s sys 0.002s 04:10:46.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:46.991434 4735 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.884s user 0.000s sys 0.002s 04:10:47.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.032404 4735 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:47.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.215798 4735 rpc_server.cc:205] RPC server started. Bound to: 127.4.61.125:46275 04:10:47.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.219812 4735 webserver.cc:374] Webserver started at http://127.4.61.125:35773/ using document root and password file 04:10:47.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.221009 4735 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/info.pb 04:10:47.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.226104 4341 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4735 04:10:47.226 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:47.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.234685 4818 sys_catalog.cc:188] Verifying existing consensus state 04:10:47.241 [DEBUG - New I/O worker #65] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:10:47.242 [DEBUG - New I/O worker #65] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc 04:10:47.244 [DEBUG - New I/O worker #65] (SecurityUtil.java:118) Logged in as subject: test-user@KRBTEST.COM 04:10:47.245 [ERROR - New I/O worker #65] (SecurityContext.java:244) Attempted to refresh Kerberos credentials from ticket cache but found that the new Kerberos principal test-user@KRBTEST.COM did not match the original principal test-admin@KRBTEST.COM. Ignoring. 04:10:47.246 [DEBUG - New I/O worker #65] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.4.61.124 04:10:47.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.254138 4818 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Bootstrap starting. 04:10:47.258 [DEBUG - New I/O worker #64] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.4.61.126 04:10:47.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.61.124@KRBTEST.COM 04:10:47.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.61.126@KRBTEST.COM 04:10:47.278 [DEBUG - New I/O worker #66] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.4.61.125 04:10:47.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.61.125@KRBTEST.COM 04:10:47.295 [DEBUG - New I/O worker #64] (Negotiator.java:764) Authenticated connection [id: 0x70fc0d51, /127.0.0.1:56668 => /127.4.61.126:46743] using SASL/GSSAPI 04:10:47.297 [DEBUG - New I/O worker #65] (Negotiator.java:764) Authenticated connection [id: 0xe52b89f8, /127.0.0.1:35596 => /127.4.61.124:36431] using SASL/GSSAPI 04:10:47.300 [DEBUG - New I/O worker #64] (AsyncKuduClient.java:2322) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 04:10:47.300 [DEBUG - New I/O worker #64] (AsyncKuduClient.java:1296) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=74), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.4.61.126:46743, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.4.61.124:36431, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.4.61.125:46275, [68ms] Sub RPC ConnectToMaster: received response from server master-127.4.61.126:46743: OK, deferred=null) after lookup 04:10:47.301 [DEBUG - New I/O worker #66] (Negotiator.java:764) Authenticated connection [id: 0x8c1c86df, /127.0.0.1:38864 => /127.4.61.125:46275] using SASL/GSSAPI 04:10:47.313 [INFO - New I/O worker #66] (ConnectToCluster.java:361) Unable to connect to master 127.4.61.125:46275: Catalog manager is not initialized. State: Starting 04:10:47.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608643, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.126@KRBTEST.COM for kudu/127.4.61.125@KRBTEST.COM 04:10:47.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.325872 4818 log.cc:525] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Log is configured to *not* fsync() on all Append() calls 04:10:47.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.340265 4818 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:47.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.340926 4818 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Bootstrap complete. 04:10:47.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.353888 4818 raft_consensus.cc:340] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:47.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.359133 4818 raft_consensus.cc:705] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1358655cf490440fbff9f81ab4ea9168, State: Initialized, Role: FOLLOWER 04:10:47.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.359827 4818 consensus_queue.cc:229] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:47.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.363705 4818 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:47.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.364212 4818 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: FOLLOWER 04:10:47.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.365475 4818 sys_catalog.cc:427] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: configured and running, proceeding with master startup. 04:10:47.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.364014 4825 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:47.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.434962 4795 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 1 FOLLOWER]: Advancing to term 2 04:10:47.435 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.435434 4825 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: FOLLOWER 04:10:47.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.455965 4829 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:47.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.457029 4825 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1846f7dc19401ebfba50d7f54975fe. Latest consensus state: current_term: 2 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:47.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.457973 4825 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: FOLLOWER 04:10:47.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.458015 4829 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: FOLLOWER 04:10:47.466 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.466492 4826 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: acquiring CA information for follower catalog manager: success 04:10:47.468 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.468514 4826 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:47.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.815490 4760 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:47.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:47.816329 4760 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: LEADER 04:10:50.354 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.4.61.126:46743 04:10:50.355 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:10:50.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.355633 4341 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4605 04:10:50.374 [DEBUG - New I/O worker #64] (Connection.java:691) [peer master-127.4.61.126:46743(127.4.61.126:46743)] cleaning up while in state READY due to: connection disconnected 04:10:50.374 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:50.374 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.4.61.124:36431 04:10:50.374 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:10:50.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.375233 4341 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4668 04:10:50.393 [DEBUG - New I/O worker #65] (Connection.java:691) [peer master-127.4.61.124:36431(127.4.61.124:36431)] cleaning up while in state READY due to: connection disconnected 04:10:50.393 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:50.393 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.4.61.125:46275 04:10:50.393 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:10:50.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.394526 4341 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4735 04:10:50.413 [DEBUG - New I/O worker #66] (Connection.java:691) [peer master-127.4.61.125:46275(127.4.61.125:46275)] cleaning up while in state READY due to: connection disconnected 04:10:50.414 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:50.414 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.4.61.126:46743 04:10:50.414 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:10:50.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.415482 4341 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:50.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:50.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:50.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data 04:10:50.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:50.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.126:46743 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.126 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=35121 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/logs 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/info.pb 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.126.keytab 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.126 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:50.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:50.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:50.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:50.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:50.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:50.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:50.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:50.562465 4834 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:50.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:50.562798 4834 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.4.61.126 04:10:50.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:50.563026 4834 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:50.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:50.574765 4834 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:50.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:50.574904 4834 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:50.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:50.575012 4834 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:50.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:50.575134 4834 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:50.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:50.575255 4834 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:50.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:50.575378 4834 flags.cc:404] Enabled experimental flag: --principal=kudu/127.4.61.126 04:10:50.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.593664 4834 master_main.cc:98] Master server non-default flags: 04:10:50.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.126:46743 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.126.keytab 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.126 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/info.pb 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.126 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=35121 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:50.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/logs 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:50.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:50.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.607800 4834 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 162016us 04:10:50.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:50.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.610021 4834 init.cc:359] Logged in from keytab as kudu/127.4.61.126@KRBTEST.COM (short username kudu) 04:10:50.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.612766 4834 fs_manager.cc:263] Metadata directory not provided 04:10:50.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.612895 4834 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:50.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.617853 4834 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s 04:10:50.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.618505 4834 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:50.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.618995 4834 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:50.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.625934 4834 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.626060 4834 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "ac1846f7dc19401ebfba50d7f54975fe" 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:36 on dist-test-slave-dist-test-slave-fvsv" 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:50.627388 4834 fs_report.cc:352] FS layout report 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/data 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:50.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:50.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:50.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:50.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:50.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:50.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:51.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:51.503851 4834 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.853s user 0.294s sys 0.558s 04:10:51.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:51.504072 4834 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.853s user 0.294s sys 0.558s 04:10:51.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.532603 4834 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:51.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.605463 4834 rpc_server.cc:205] RPC server started. Bound to: 127.4.61.126:46743 04:10:51.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.608709 4834 webserver.cc:374] Webserver started at http://127.4.61.126:35121/ using document root and password file 04:10:51.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.609937 4834 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/info.pb 04:10:51.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.611302 4341 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4834 04:10:51.611 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:51.611 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.4.61.124:36431 04:10:51.612 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:10:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.613189 4341 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data 04:10:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.124:36431 04:10:51.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.124 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=38787 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/logs 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/info.pb 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.124.keytab 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.124 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:51.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:51.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:51.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:51.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:51.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.618878 4896 sys_catalog.cc:188] Verifying existing consensus state 04:10:51.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.622637 4896 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Bootstrap starting. 04:10:51.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.650224 4896 log.cc:525] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Log is configured to *not* fsync() on all Append() calls 04:10:51.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.663228 4896 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:51.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.663900 4896 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Bootstrap complete. 04:10:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.684082 4896 raft_consensus.cc:340] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:51.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.689390 4896 raft_consensus.cc:705] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1846f7dc19401ebfba50d7f54975fe, State: Initialized, Role: FOLLOWER 04:10:51.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.690151 4896 consensus_queue.cc:229] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:51.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.692392 4899 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:51.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.692914 4899 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: FOLLOWER 04:10:51.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.693300 4896 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:51.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.693708 4896 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: FOLLOWER 04:10:51.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.694540 4896 sys_catalog.cc:427] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: configured and running, proceeding with master startup. 04:10:51.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.696038 4900 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:10:51.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:51.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:51.763339 4897 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:51.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:51.763643 4897 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.4.61.124 04:10:51.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:51.763875 4897 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:51.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:51.775343 4897 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:51.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:51.775496 4897 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:51.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:51.775614 4897 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:51.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:51.775707 4897 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:51.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:51.775817 4897 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:51.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:51.775936 4897 flags.cc:404] Enabled experimental flag: --principal=kudu/127.4.61.124 04:10:51.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.795302 4897 master_main.cc:98] Master server non-default flags: 04:10:51.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data 04:10:51.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:51.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:51.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:51.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 04:10:51.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:51.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:51.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.124:36431 04:10:51.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:51.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:51.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.124.keytab 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.124 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/info.pb 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.124 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=38787 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/logs 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:51.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:51.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.809473 4897 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 162516us 04:10:51.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:51 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608651, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:51.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.811740 4897 init.cc:359] Logged in from keytab as kudu/127.4.61.124@KRBTEST.COM (short username kudu) 04:10:51.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.815460 4897 fs_manager.cc:263] Metadata directory not provided 04:10:51.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.815585 4897 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:51.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.820521 4897 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s 04:10:51.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.821149 4897 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:51.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.821620 4897 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:51.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.828248 4897 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:10:51.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.828371 4897 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:51.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "9a549346b3a942b28912139226b10196" 04:10:51.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:38 on dist-test-slave-dist-test-slave-fvsv" 04:10:51.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:51.829604 4897 fs_report.cc:352] FS layout report 04:10:51.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:51.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:51.829 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:51.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/data 04:10:51.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:51.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:51.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:51.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:51.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:51.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:51.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:51.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:51.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:51.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:52.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:52.633344 4897 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.774s user 0.001s sys 0.001s 04:10:52.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:52.633623 4897 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.774s user 0.001s sys 0.001s 04:10:52.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.704530 4900 catalog_manager.cc:1120] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: acquiring CA information for follower catalog manager: success 04:10:52.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.707418 4900 catalog_manager.cc:1148] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:52.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.724489 4897 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:52.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.804042 4897 rpc_server.cc:205] RPC server started. Bound to: 127.4.61.124:36431 04:10:52.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.806970 4897 webserver.cc:374] Webserver started at http://127.4.61.124:38787/ using document root and password file 04:10:52.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.808001 4897 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/info.pb 04:10:52.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.809984 4341 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4897 04:10:52.810 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:52.810 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.4.61.125:46275 04:10:52.810 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.811908 4341 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.125:46275 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.125 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=35773 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:52.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/logs 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/info.pb 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.125.keytab 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.125 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:52.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.816792 4963 sys_catalog.cc:188] Verifying existing consensus state 04:10:52.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.824753 4963 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Bootstrap starting. 04:10:52.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.868057 4963 log.cc:525] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Log is configured to *not* fsync() on all Append() calls 04:10:52.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.879696 4963 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:52.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.880498 4963 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Bootstrap complete. 04:10:52.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.917645 4963 raft_consensus.cc:340] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:52.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.925757 4963 raft_consensus.cc:705] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a549346b3a942b28912139226b10196, State: Initialized, Role: FOLLOWER 04:10:52.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.926568 4963 consensus_queue.cc:229] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:52.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.931910 4963 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:52.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.932551 4963 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:10:52.934 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.933961 4963 sys_catalog.cc:427] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: configured and running, proceeding with master startup. 04:10:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:52.932255 4966 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:53.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.041393 4966 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:10:53.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.053164 4967 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: acquiring CA information for follower catalog manager: success 04:10:53.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.056104 4967 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:53.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:53.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:53.077181 4964 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:53.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:53.077652 4964 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.4.61.125 04:10:53.078 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:53.078014 4964 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:53.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:53.097045 4964 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:53.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:53.097255 4964 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:53.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:53.097432 4964 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:53.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:53.097601 4964 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:53.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:53.097786 4964 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:53.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:53.097991 4964 flags.cc:404] Enabled experimental flag: --principal=kudu/127.4.61.125 04:10:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.130197 4964 master_main.cc:98] Master server non-default flags: 04:10:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data 04:10:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 04:10:53.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.125:46275 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.125.keytab 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.125 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/info.pb 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.125 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=35773 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:53.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/logs 04:10:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:53.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:53.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.154384 4964 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 163516us 04:10:53.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:53 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608653, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:53.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.158010 4964 init.cc:359] Logged in from keytab as kudu/127.4.61.125@KRBTEST.COM (short username kudu) 04:10:53.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.181731 4964 fs_manager.cc:263] Metadata directory not provided 04:10:53.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.181943 4964 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:53.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.189306 4964 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s 04:10:53.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.190255 4964 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:53.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.190958 4964 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:53.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.204730 4964 fs_manager.cc:419] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s 04:10:53.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.204913 4964 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:53.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1358655cf490440fbff9f81ab4ea9168" 04:10:53.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:37 on dist-test-slave-dist-test-slave-fvsv" 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.206789 4964 fs_report.cc:352] FS layout report 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/data 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:53.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:53.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.157148 4973 raft_consensus.cc:472] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:10:53.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.266062 4973 raft_consensus.cc:494] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:53.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.275457 4973 leader_election.cc:296] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275), 9a549346b3a942b28912139226b10196 (127.4.61.124:36431) 04:10:53.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:53.278910 4847 leader_election.cc:341] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275): Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111) 04:10:53.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.291870 4940 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac1846f7dc19401ebfba50d7f54975fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9a549346b3a942b28912139226b10196" is_pre_election: true 04:10:53.293 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.292722 4940 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac1846f7dc19401ebfba50d7f54975fe in term 2. 04:10:53.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.294152 4843 leader_election.cc:310] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a549346b3a942b28912139226b10196, ac1846f7dc19401ebfba50d7f54975fe; no voters: 1358655cf490440fbff9f81ab4ea9168 04:10:53.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.294962 4973 raft_consensus.cc:2644] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 2 FOLLOWER]: Leader pre-election won for term 3 04:10:53.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.295271 4973 raft_consensus.cc:472] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:10:53.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.295521 4973 raft_consensus.cc:2890] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 2 FOLLOWER]: Advancing to term 3 04:10:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.302564 4973 raft_consensus.cc:494] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:53.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:53.305016 4847 leader_election.cc:341] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275): Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111) 04:10:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.305842 4940 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac1846f7dc19401ebfba50d7f54975fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "9a549346b3a942b28912139226b10196" 04:10:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.306481 4940 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 2 FOLLOWER]: Advancing to term 3 04:10:53.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.313285 4940 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac1846f7dc19401ebfba50d7f54975fe in term 3. 04:10:53.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.314391 4843 leader_election.cc:310] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a549346b3a942b28912139226b10196, ac1846f7dc19401ebfba50d7f54975fe; no voters: 1358655cf490440fbff9f81ab4ea9168 04:10:53.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.316738 4973 leader_election.cc:296] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 3 election: Requested vote from peers 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275), 9a549346b3a942b28912139226b10196 (127.4.61.124:36431) 04:10:53.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.316992 4985 raft_consensus.cc:2644] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 3 FOLLOWER]: Leader election won for term 3 04:10:53.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.317555 4985 raft_consensus.cc:667] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 3 LEADER]: Becoming Leader. State: Replica: ac1846f7dc19401ebfba50d7f54975fe, State: Running, Role: LEADER 04:10:53.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.318229 4985 consensus_queue.cc:206] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:53.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.323801 4973 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1846f7dc19401ebfba50d7f54975fe. Latest consensus state: current_term: 3 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:53.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.324702 4973 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: LEADER 04:10:53.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.327461 4987 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:10:53.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.332554 4987 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:10:53.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.335337 4987 catalog_manager.cc:1084] Loading token signing keys... 04:10:53.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.337644 4987 catalog_manager.cc:4361] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Loaded TSK: 0 04:10:53.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.701772 4940 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 3 FOLLOWER]: Refusing update from remote peer ac1846f7dc19401ebfba50d7f54975fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) 04:10:53.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.703603 4973 consensus_queue.cc:984] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s 04:10:53.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.716748 4988 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1846f7dc19401ebfba50d7f54975fe. Latest consensus state: current_term: 3 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:53.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.717864 4988 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:10:53.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:53.735517 4847 consensus_peers.cc:458] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe -> Peer 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275): Couldn't send request to peer 1358655cf490440fbff9f81ab4ea9168. Status: Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:10:53.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.736469 4985 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:53.737 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.737366 4985 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: LEADER 04:10:53.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.748208 4988 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:53.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:53.748972 4988 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:10:54.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:54.065697 4964 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.822s user 0.002s sys 0.001s 04:10:54.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:54.065986 4964 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.822s user 0.002s sys 0.001s 04:10:54.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.152671 4964 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:54.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.403861 4964 rpc_server.cc:205] RPC server started. Bound to: 127.4.61.125:46275 04:10:54.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.407801 4964 webserver.cc:374] Webserver started at http://127.4.61.125:35773/ using document root and password file 04:10:54.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.409231 4964 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/info.pb 04:10:54.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.419293 4341 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 4964 04:10:54.419 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:54.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.422296 5041 sys_catalog.cc:188] Verifying existing consensus state 04:10:54.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.437759 5041 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Bootstrap starting. 04:10:54.440 [DEBUG - New I/O worker #67] (Negotiator.java:764) Authenticated connection [id: 0xd88ddde2, /127.0.0.1:56686 => /127.4.61.126:46743] using TOKEN/null 04:10:54.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.507997 5041 log.cc:525] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Log is configured to *not* fsync() on all Append() calls 04:10:54.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.517016 5041 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:54.517 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.517604 5041 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Bootstrap complete. 04:10:54.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.537763 5041 raft_consensus.cc:340] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:54.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.542963 5041 raft_consensus.cc:705] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1358655cf490440fbff9f81ab4ea9168, State: Initialized, Role: FOLLOWER 04:10:54.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.543771 5041 consensus_queue.cc:229] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:54.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.547776 5041 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:54.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.548265 5041 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: FOLLOWER 04:10:54.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.549341 5041 sys_catalog.cc:427] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: configured and running, proceeding with master startup. 04:10:54.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.548208 5044 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:54.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.628937 5044 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: FOLLOWER 04:10:54.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.641153 5045 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: acquiring CA information for follower catalog manager: success 04:10:54.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.644623 5045 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:54.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:54 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.126@KRBTEST.COM for kudu/127.4.61.125@KRBTEST.COM 04:10:54.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.662205 5018 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [term 2 FOLLOWER]: Advancing to term 3 04:10:54.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.679270 5044 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1846f7dc19401ebfba50d7f54975fe. Latest consensus state: current_term: 3 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:54.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.679987 5044 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: FOLLOWER 04:10:54.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.680389 5050 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:54.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.681150 5050 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168 [sys.catalog]: This master's current role is: FOLLOWER 04:10:54.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.989955 4973 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:54.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:54.990730 4973 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: LEADER 04:10:57.494 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.4.61.126:46743 04:10:57.494 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:10:57.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.495453 4341 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4834 04:10:57.514 [DEBUG - New I/O worker #67] (Connection.java:691) [peer master-127.4.61.126:46743(127.4.61.126:46743)] cleaning up while in state READY due to: connection disconnected 04:10:57.515 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:57.515 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.4.61.124:36431 04:10:57.515 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:10:57.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.516150 4341 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4897 04:10:57.534 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:57.534 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.4.61.125:46275 04:10:57.534 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:10:57.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.535189 4341 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 4964 04:10:57.557 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:57.557 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.4.61.126:46743 04:10:57.557 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.558799 4341 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.126:46743 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.126 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=35121 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/logs 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/info.pb 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.126.keytab 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.126 04:10:57.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:57.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:57.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:57.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:57.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:57.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:57.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:57.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:57.708643 5055 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:57.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:57.709004 5055 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.4.61.126 04:10:57.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:57.709342 5055 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:57.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:57.720726 5055 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:57.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:57.720885 5055 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:57.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:57.721033 5055 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:57.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:57.721163 5055 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:57.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:57.721336 5055 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:57.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:57.721513 5055 flags.cc:404] Enabled experimental flag: --principal=kudu/127.4.61.126 04:10:57.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.739600 5055 master_main.cc:98] Master server non-default flags: 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.126:46743 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.126.keytab 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.126 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/info.pb 04:10:57.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.126 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=35121 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/logs 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:57.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:57.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.754415 5055 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 165516us 04:10:57.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:57 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608657, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:57.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.756762 5055 init.cc:359] Logged in from keytab as kudu/127.4.61.126@KRBTEST.COM (short username kudu) 04:10:57.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.759485 5055 fs_manager.cc:263] Metadata directory not provided 04:10:57.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.759613 5055 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:57.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.764556 5055 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s 04:10:57.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.765204 5055 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:57.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.765693 5055 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:57.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.772294 5055 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s 04:10:57.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.772419 5055 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:57.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "ac1846f7dc19401ebfba50d7f54975fe" 04:10:57.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:36 on dist-test-slave-dist-test-slave-fvsv" 04:10:57.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:57.773722 5055 fs_report.cc:352] FS layout report 04:10:57.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/wal 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/data 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:57.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:58.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:58.647485 5055 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.850s user 0.353s sys 0.469s 04:10:58.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:58.647708 5055 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.850s user 0.353s sys 0.469s 04:10:58.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.705025 5055 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:58.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.770135 5055 rpc_server.cc:205] RPC server started. Bound to: 127.4.61.126:46743 04:10:58.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.772486 5055 webserver.cc:374] Webserver started at http://127.4.61.126:35121/ using document root and password file 04:10:58.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.773408 5055 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-0/data/info.pb 04:10:58.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.777778 4341 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 5055 04:10:58.778 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:58.778 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.4.61.124:36431 04:10:58.778 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:10:58.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.779340 4341 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:58.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:58.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:58.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data 04:10:58.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:58.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:58.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:58.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:58.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.124:36431 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.124 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=38787 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/logs 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/info.pb 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.124.keytab 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.124 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:58.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:58.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:58.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:58.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:58.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.780982 5117 sys_catalog.cc:188] Verifying existing consensus state 04:10:58.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.784977 5117 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Bootstrap starting. 04:10:58.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.831709 5117 log.cc:525] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Log is configured to *not* fsync() on all Append() calls 04:10:58.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.840764 5117 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:58.841 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.841358 5117 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Bootstrap complete. 04:10:58.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.861475 5117 raft_consensus.cc:340] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:58.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.866708 5117 raft_consensus.cc:705] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1846f7dc19401ebfba50d7f54975fe, State: Initialized, Role: FOLLOWER 04:10:58.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.867444 5117 consensus_queue.cc:229] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:10:58.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.869578 5120 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:58.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.870187 5120 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: FOLLOWER 04:10:58.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.870677 5117 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:10:58.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.871201 5117 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: FOLLOWER 04:10:58.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.872041 5117 sys_catalog.cc:427] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: configured and running, proceeding with master startup. 04:10:58.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.890681 5121 catalog_manager.cc:1120] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: acquiring CA information for follower catalog manager: success 04:10:58.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:58.896801 5121 catalog_manager.cc:1148] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:10:58.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:10:58.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:58.989126 5118 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:10:58.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:58.989498 5118 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.4.61.124 04:10:58.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:58.989821 5118 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:10:59.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:59.001163 5118 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:10:59.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:59.001315 5118 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:10:59.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:59.001461 5118 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:10:59.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:59.001591 5118 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:10:59.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:59.001770 5118 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:10:59.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:59.001947 5118 flags.cc:404] Enabled experimental flag: --principal=kudu/127.4.61.124 04:10:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.021186 5118 master_main.cc:98] Master server non-default flags: 04:10:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data 04:10:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 04:10:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:59.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.124:36431 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.124.keytab 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.124 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/info.pb 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.124 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=38787 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:59.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/logs 04:10:59.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:59.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:10:59.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:10:59.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:10:59.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:10:59.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:10:59.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:10:59.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:10:59.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:10:59.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.035596 5118 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 166516us 04:10:59.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:10:59 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608659, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:10:59.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.038027 5118 init.cc:359] Logged in from keytab as kudu/127.4.61.124@KRBTEST.COM (short username kudu) 04:10:59.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.047767 5118 fs_manager.cc:263] Metadata directory not provided 04:10:59.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.047920 5118 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:10:59.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.056493 5118 fs_manager.cc:399] Time spent opening directory manager: real 0.007s user 0.000s sys 0.008s 04:10:59.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.057201 5118 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:10:59.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.057759 5118 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:10:59.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.068367 5118 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s 04:10:59.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.068521 5118 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:59.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "9a549346b3a942b28912139226b10196" 04:10:59.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:38 on dist-test-slave-dist-test-slave-fvsv" 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.069845 5118 fs_report.cc:352] FS layout report 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/wal 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/data 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:10:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:10:59.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:59.752775 5118 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.656s user 0.238s sys 0.415s 04:10:59.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:10:59.753304 5118 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.656s user 0.239s sys 0.415s 04:10:59.788 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.788450 5118 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:10:59.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.896222 5118 rpc_server.cc:205] RPC server started. Bound to: 127.4.61.124:36431 04:10:59.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.899410 5118 webserver.cc:374] Webserver started at http://127.4.61.124:38787/ using document root and password file 04:10:59.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.900624 5118 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-2/data/info.pb 04:10:59.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.908578 5184 sys_catalog.cc:188] Verifying existing consensus state 04:10:59.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.910698 4341 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 5118 04:10:59.911 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:10:59.911 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.4.61.125:46275 04:10:59.911 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:10:59.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.912684 4341 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.125:46275 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.125 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=35773 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/logs 04:10:59.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/info.pb 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.125.keytab 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.125 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:10:59.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.913328 5184 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Bootstrap starting. 04:10:59.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.972935 5184 log.cc:525] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Log is configured to *not* fsync() on all Append() calls 04:10:59.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.989719 5184 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:10:59.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:10:59.991101 5184 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: Bootstrap complete. 04:11:00.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.021908 5184 raft_consensus.cc:340] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:11:00.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.029891 5184 raft_consensus.cc:705] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a549346b3a942b28912139226b10196, State: Initialized, Role: FOLLOWER 04:11:00.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.030712 5184 consensus_queue.cc:229] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:11:00.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.033701 5187 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:11:00.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.034418 5187 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:11:00.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.034873 5184 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:11:00.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.035488 5184 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:11:00.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.036618 5184 sys_catalog.cc:427] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: configured and running, proceeding with master startup. 04:11:00.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.038616 5188 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:11:00.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:00.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:00.113111 5185 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:00.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:00.113449 5185 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.4.61.125 04:11:00.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:00.113673 5185 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:00.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:00.124529 5185 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:00.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:00.124670 5185 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:00.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:00.124775 5185 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:00.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:00.124873 5185 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:00.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:00.124982 5185 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:00.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:00.125099 5185 flags.cc:404] Enabled experimental flag: --principal=kudu/127.4.61.125 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.150148 5185 master_main.cc:98] Master server non-default flags: 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.4.61.126:46743,127.4.61.125:46275,127.4.61.124:36431 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.4.61.125:46275 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/kudu_127.4.61.125.keytab 04:11:00.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.4.61.125 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/info.pb 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.4.61.125 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=35773 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/logs 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:00.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:00.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.173655 5185 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 167016us 04:11:00.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:00 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608660, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:00.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.176646 5185 init.cc:359] Logged in from keytab as kudu/127.4.61.125@KRBTEST.COM (short username kudu) 04:11:00.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.180088 5185 fs_manager.cc:263] Metadata directory not provided 04:11:00.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.180263 5185 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:00.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.187021 5185 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s 04:11:00.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.187968 5185 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:00.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.188653 5185 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:00.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.199635 5185 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:11:00.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.199815 5185 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:11:00.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1358655cf490440fbff9f81ab4ea9168" 04:11:00.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:10:37 on dist-test-slave-dist-test-slave-fvsv" 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.201651 5185 fs_report.cc:352] FS layout report 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/wal 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/data 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:00.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:00.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.416250 5201 raft_consensus.cc:472] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:11:00.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.416985 5201 raft_consensus.cc:494] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:11:00.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:00.423362 5068 leader_election.cc:341] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275): Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111) 04:11:00.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:00 dist-test-slave-dist-test-slave-fvsv krb5kdc[4350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608657, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.61.126@KRBTEST.COM for kudu/127.4.61.124@KRBTEST.COM 04:11:00.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.471464 5161 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac1846f7dc19401ebfba50d7f54975fe" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9a549346b3a942b28912139226b10196" is_pre_election: true 04:11:00.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.472306 5161 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac1846f7dc19401ebfba50d7f54975fe in term 3. 04:11:00.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.474030 5064 leader_election.cc:310] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a549346b3a942b28912139226b10196, ac1846f7dc19401ebfba50d7f54975fe; no voters: 1358655cf490440fbff9f81ab4ea9168 04:11:00.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.427217 5201 leader_election.cc:296] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275), 9a549346b3a942b28912139226b10196 (127.4.61.124:36431) 04:11:00.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.560087 5201 raft_consensus.cc:2644] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 3 FOLLOWER]: Leader pre-election won for term 4 04:11:00.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.560350 5201 raft_consensus.cc:472] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:11:00.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.560572 5201 raft_consensus.cc:2890] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 3 FOLLOWER]: Advancing to term 4 04:11:00.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:00.564038 5064 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.61.124:36431, user_credentials={real_user=slave}} blocked reactor thread for 90190.8us 04:11:00.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.568243 5201 raft_consensus.cc:494] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:11:00.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.571408 5161 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac1846f7dc19401ebfba50d7f54975fe" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "9a549346b3a942b28912139226b10196" 04:11:00.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:00.571590 5068 leader_election.cc:341] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275): Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111) 04:11:00.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.571907 5161 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 3 FOLLOWER]: Advancing to term 4 04:11:00.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.579073 5161 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac1846f7dc19401ebfba50d7f54975fe in term 4. 04:11:00.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.580109 5064 leader_election.cc:310] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9a549346b3a942b28912139226b10196, ac1846f7dc19401ebfba50d7f54975fe; no voters: 1358655cf490440fbff9f81ab4ea9168 04:11:00.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.582830 5205 raft_consensus.cc:2644] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 4 FOLLOWER]: Leader election won for term 4 04:11:00.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.584324 5201 leader_election.cc:296] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [CANDIDATE]: Term 4 election: Requested vote from peers 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275), 9a549346b3a942b28912139226b10196 (127.4.61.124:36431) 04:11:00.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.587726 5205 raft_consensus.cc:667] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [term 4 LEADER]: Becoming Leader. State: Replica: ac1846f7dc19401ebfba50d7f54975fe, State: Running, Role: LEADER 04:11:00.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.588433 5205 consensus_queue.cc:206] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } 04:11:00.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.597338 5201 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1846f7dc19401ebfba50d7f54975fe. Latest consensus state: current_term: 4 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:11:00.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.597961 5201 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: LEADER 04:11:00.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.604216 5208 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:11:00.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.608728 5208 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:11:00.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.615862 5208 catalog_manager.cc:1084] Loading token signing keys... 04:11:00.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.618151 5208 catalog_manager.cc:4361] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe: Loaded TSK: 0 04:11:00.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.996891 5161 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [term 4 FOLLOWER]: Refusing update from remote peer ac1846f7dc19401ebfba50d7f54975fe: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch) 04:11:00.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:00.998181 5201 consensus_queue.cc:984] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s 04:11:01.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.006064 5209 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1846f7dc19401ebfba50d7f54975fe. Latest consensus state: current_term: 4 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:11:01.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.006767 5209 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:11:01.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:01.013980 5068 consensus_peers.cc:458] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe -> Peer 1358655cf490440fbff9f81ab4ea9168 (127.4.61.125:46275): Couldn't send request to peer 1358655cf490440fbff9f81ab4ea9168. Status: Network error: Client connection negotiation failed: client connection to 127.4.61.125:46275: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:11:01.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.014789 5205 sys_catalog.cc:339] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:11:01.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.015182 5209 sys_catalog.cc:339] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "ac1846f7dc19401ebfba50d7f54975fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac1846f7dc19401ebfba50d7f54975fe" member_type: VOTER last_known_addr { host: "127.4.61.126" port: 46743 } } peers { permanent_uuid: "1358655cf490440fbff9f81ab4ea9168" member_type: VOTER last_known_addr { host: "127.4.61.125" port: 46275 } } peers { permanent_uuid: "9a549346b3a942b28912139226b10196" member_type: VOTER last_known_addr { host: "127.4.61.124" port: 36431 } } } 04:11:01.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.015645 5205 sys_catalog.cc:342] T 00000000000000000000000000000000 P ac1846f7dc19401ebfba50d7f54975fe [sys.catalog]: This master's current role is: LEADER 04:11:01.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.015825 5209 sys_catalog.cc:342] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196 [sys.catalog]: This master's current role is: FOLLOWER 04:11:01.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:01.170188 5185 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.930s user 0.316s sys 0.599s 04:11:01.170 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:01.170473 5185 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.931s user 0.316s sys 0.599s 04:11:01.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.049780 5188 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: acquiring CA information for follower catalog manager: success 04:11:01.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.186398 5188 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 9a549346b3a942b28912139226b10196: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:01.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.228509 5185 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:01.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.358785 5185 rpc_server.cc:205] RPC server started. Bound to: 127.4.61.125:46275 04:11:01.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.362138 5185 webserver.cc:374] Webserver started at http://127.4.61.125:35773/ using document root and password file 04:11:01.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.363369 5185 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/master-1/data/info.pb 04:11:01.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.366977 4341 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 5185 04:11:01.367 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:01.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:01.381095 5264 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56698: token expired 04:11:01.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.384090 5263 sys_catalog.cc:188] Verifying existing consensus state 04:11:01.384 [DEBUG - New I/O worker #68] (Negotiator.java:291) peer /127.4.61.126:46743 sent connection negotiation error: Not authorized: token expired 04:11:01.385 [DEBUG - New I/O worker #68] (Connection.java:691) [peer master-127.4.61.126:46743(127.4.61.126:46743)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 04:11:01.391 [DEBUG - New I/O worker #70] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:11:01.391 [DEBUG - New I/O worker #70] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster8687394327169445663/krb5kdc/krb5cc 04:11:01.393 [DEBUG - New I/O worker #70] (SecurityUtil.java:121) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 04:11:01.393 [WARN - New I/O worker #70] (SecurityContext.java:230) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache 04:11:01.393 [DEBUG - New I/O worker #69] (Connection.java:691) [peer master-127.4.61.126:46743(127.4.61.126:46743)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because this connection will be used to acquire a new token and therefore requires primary credentials 04:11:01.394 [INFO - New I/O worker #69] (ConnectToCluster.java:361) Unable to connect to master 127.4.61.126:46743: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because this connection will be used to acquire a new token and therefore requires primary credentials 04:11:01.394 [DEBUG - New I/O worker #70] (Connection.java:691) [peer master-127.4.61.124:36431(127.4.61.124:36431)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because this connection will be used to acquire a new token and therefore requires primary credentials 04:11:01.394 [INFO - New I/O worker #70] (ConnectToCluster.java:361) Unable to connect to master 127.4.61.124:36431: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because this connection will be used to acquire a new token and therefore requires primary credentials 04:11:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:01.396508 5265 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.388490 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:35628 04:11:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390456 (+ 1966us) server_negotiation.cc:184] Beginning negotiation 04:11:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390477 (+ 21us) server_negotiation.cc:373] Waiting for connection header 04:11:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390518 (+ 41us) server_negotiation.cc:381] Connection header received 04:11:01.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390652 (+ 134us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390673 (+ 21us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390802 (+ 129us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.393704 (+ 2902us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.395418 (+ 1714us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35628: BlockingRecv error: recv got EOF from 127.0.0.1:35628 (error 108) 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":1646,"thread_start_us":1580,"threads_started":1} 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.394935 5263 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1358655cf490440fbff9f81ab4ea9168: Bootstrap starting. 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:01.397435 5264 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.389959 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:56700 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390594 (+ 635us) server_negotiation.cc:184] Beginning negotiation 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390618 (+ 24us) server_negotiation.cc:373] Waiting for connection header 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390656 (+ 38us) server_negotiation.cc:381] Connection header received 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390773 (+ 117us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390798 (+ 25us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:01.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390936 (+ 138us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:01.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.391126 (+ 190us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:01.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.395868 (+ 4742us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56700: BlockingRecv error: recv got EOF from 127.0.0.1:56700 (error 108) 04:11:01.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":355} 04:11:01.409 [DEBUG - New I/O worker #71] (Connection.java:691) [peer master-127.4.61.125:46275(127.4.61.125:46275)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because this connection will be used to acquire a new token and therefore requires primary credentials 04:11:01.410 [INFO - New I/O worker #71] (ConnectToCluster.java:361) Unable to connect to master 127.4.61.125:46275: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because this connection will be used to acquire a new token and therefore requires primary credentials 04:11:01.411 [DEBUG - main] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:11:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:01.412958 5266 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.390378 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:38896 04:11:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.395430 (+ 5052us) server_negotiation.cc:184] Beginning negotiation 04:11:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.395459 (+ 29us) server_negotiation.cc:373] Waiting for connection header 04:11:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.395530 (+ 71us) server_negotiation.cc:381] Connection header received 04:11:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.408372 (+ 12842us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.408424 (+ 52us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.408736 (+ 312us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.409143 (+ 407us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:01.411385 (+ 2242us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38896: BlockingRecv error: recv got EOF from 127.0.0.1:38896 (error 108) 04:11:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":4658,"thread_start_us":4074,"threads_started":1} 04:11:01.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.413460 4341 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 5055 04:11:01.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.447715 4341 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 5185 04:11:01.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:01.460327 4341 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 5118 04:11:02.502 [INFO - main] (RetryRule.java:136) Creating RetryStatement with result reporter and retry count of 0 (no retries) .04:11:02.502 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskusvpEt/test-tmp 04:11:02.502 [INFO - main] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin 04:11:02.502 [INFO - main] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskusvpEt/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb] 04:11:02.503 [DEBUG - main] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 3 enable_kerberos: true cluster_root: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440" hms_mode: NONE mini_kdc_options { } } 04:11:02.680 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:11:02.680 [DEBUG - main] (MiniKuduCluster.java:155) Request: start_cluster { } 04:11:02.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Loading random data 04:11:02.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Initializing database '/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/principal' for realm 'KRBTEST.COM', 04:11:02.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) master key name 'K/M@KRBTEST.COM' 04:11:02.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:02 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): setting up network... 04:11:02.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:02 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) 04:11:02.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked 04:11:02.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 04:11:02.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:02 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): skipping unrecognized local address family 17 04:11:02.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:02 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): skipping unrecognized local address family 17 04:11:02.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:02 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): set up 1 sockets 04:11:02.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:02 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): commencing operation 04:11:02.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: starting... 04:11:03.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:03.259680 5269 mini_kdc.cc:119] Time spent starting KDC: real 0.552s user 0.004s sys 0.014s 04:11:03.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy 04:11:03.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:03.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-admin@KRBTEST.COM" created. 04:11:03.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy 04:11:03.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:03.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-user@KRBTEST.COM" created. 04:11:03.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy 04:11:03.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:03.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "joe-interloper@KRBTEST.COM" created. 04:11:03.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:03.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/test-user.keytab. 04:11:03.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/test-user.keytab. 04:11:03.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/test-user.keytab. 04:11:03.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/test-user.keytab. 04:11:03.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:03 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:03.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:11:03.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.5.37.126@KRBTEST.COM; defaulting to no policy 04:11:03.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:03.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.5.37.126@KRBTEST.COM" created. 04:11:03.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:03.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.126.keytab. 04:11:03.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.126.keytab. 04:11:03.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.126.keytab. 04:11:03.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.126.keytab. 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.416945 5269 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/wal 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/data 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.37.126:45983 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.37.126 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/logs 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/data/info.pb 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.126.keytab 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.37.126 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:03.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:03.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:03.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:03.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.5.37.126:45983,127.5.37.125:34095,127.5.37.124:34225 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:03.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:03.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:03.563544 5289 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:03.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:03.563894 5289 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.5.37.126 04:11:03.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:03.564128 5289 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:03.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:03.575003 5289 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:03.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:03.575183 5289 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:03.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:03.575278 5289 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:03.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:03.575387 5289 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:03.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:03.575510 5289 flags.cc:404] Enabled experimental flag: --principal=kudu/127.5.37.126 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.593911 5289 master_main.cc:98] Master server non-default flags: 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/data 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/wal 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.5.37.126:45983,127.5.37.125:34095,127.5.37.124:34225 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.37.126:45983 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.126.keytab 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.37.126 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/data/info.pb 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.37.126 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:03.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:03.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/logs 04:11:03.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:03.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:03.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:03.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:03.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:03.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:03.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:03.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:03.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:03.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.608013 5289 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 168516us 04:11:03.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:03 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:03.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.610253 5289 init.cc:359] Logged in from keytab as kudu/127.5.37.126@KRBTEST.COM (short username kudu) 04:11:03.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.613538 5289 fs_manager.cc:263] Metadata directory not provided 04:11:03.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.613667 5289 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:03.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.613893 5289 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:03.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.613997 5289 server_base.cc:449] Attempting to create new FS layout instead 04:11:03.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.616940 5289 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/data/instance: 04:11:03.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" 04:11:03.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:03 on dist-test-slave-dist-test-slave-fvsv" 04:11:03.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.617657 5289 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/wal/instance: 04:11:03.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" 04:11:03.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:03 on dist-test-slave-dist-test-slave-fvsv" 04:11:03.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.626418 5289 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s 04:11:03.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.627645 5289 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:03.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.628162 5289 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:03.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.635265 5289 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:11:03.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.635385 5289 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/wal 04:11:03.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" 04:11:03.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:03 on dist-test-slave-dist-test-slave-fvsv" 04:11:03.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:03.635597 5289 fs_report.cc:352] FS layout report 04:11:03.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:03.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/wal 04:11:03.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/wal 04:11:03.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/data/data 04:11:03.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:03.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:03.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:03.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:03.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:03.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:03.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:03.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:03.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:03.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:04.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.511739 5289 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.851s user 0.301s sys 0.549s 04:11:04.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.511937 5289 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.852s user 0.301s sys 0.549s 04:11:04.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.557299 5289 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:04.631 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.631741 5289 rpc_server.cc:205] RPC server started. Bound to: 127.5.37.126:45983 04:11:04.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.634097 5289 webserver.cc:374] Webserver started at http://127.5.37.126:44069/ using document root and password file 04:11:04.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.634980 5289 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-0/data/info.pb 04:11:04.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.638190 5351 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:04.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.642524 5269 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 5289 04:11:04.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.652350 5351 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } has no permanent_uuid. Determining permanent_uuid... 04:11:04.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.5.37.125@KRBTEST.COM; defaulting to no policy 04:11:04.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:04.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.5.37.125@KRBTEST.COM" created. 04:11:04.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:04 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.126@KRBTEST.COM for kudu/127.5.37.126@KRBTEST.COM 04:11:04.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:04.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.125.keytab. 04:11:04.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.125.keytab. 04:11:04.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.125 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.125.keytab. 04:11:04.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.125 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.125.keytab. 04:11:04.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.681016 5351 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } has no permanent_uuid. Determining permanent_uuid... 04:11:04.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.682718 5269 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/wal 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/data 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.37.125:34095 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.37.125 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/logs 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/data/info.pb 04:11:04.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.125.keytab 04:11:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.37.125 04:11:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.5.37.126:45983,127.5.37.125:34095,127.5.37.124:34225 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:04.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.682747 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.125:34095: Network error: Client connection negotiation failed: client connection to 127.5.37.125:34095: connect: Connection refused (error 111) 04:11:04.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.702121 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } attempt: 1 04:11:04.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.703760 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.125:34095: Network error: Client connection negotiation failed: client connection to 127.5.37.125:34095: connect: Connection refused (error 111) 04:11:04.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.752133 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } attempt: 2 04:11:04.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.753882 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.125:34095: Network error: Client connection negotiation failed: client connection to 127.5.37.125:34095: connect: Connection refused (error 111) 04:11:04.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:04.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.830687 5356 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:04.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.831024 5356 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.5.37.125 04:11:04.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.831305 5356 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:04.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.842226 5356 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:04.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.842380 5356 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:04.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.842474 5356 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:04.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.842587 5356 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:04.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.842708 5356 flags.cc:404] Enabled experimental flag: --principal=kudu/127.5.37.125 04:11:04.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.861163 5356 master_main.cc:98] Master server non-default flags: 04:11:04.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/data 04:11:04.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/wal 04:11:04.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:04.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.5.37.126:45983,127.5.37.125:34095,127.5.37.124:34225 04:11:04.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:04.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:04.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.37.125:34095 04:11:04.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:04.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:04.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.125.keytab 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.37.125 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/data/info.pb 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.37.125 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/logs 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:04.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:04.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.865234 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } attempt: 3 04:11:04.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:04.866829 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.125:34095: Network error: Client connection negotiation failed: client connection to 127.5.37.125:34095: connect: Connection refused (error 111) 04:11:04.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.876037 5356 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 169016us 04:11:04.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:04 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:04.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.878222 5356 init.cc:359] Logged in from keytab as kudu/127.5.37.125@KRBTEST.COM (short username kudu) 04:11:04.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.881552 5356 fs_manager.cc:263] Metadata directory not provided 04:11:04.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.881672 5356 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:04.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.881902 5356 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:04.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.882002 5356 server_base.cc:449] Attempting to create new FS layout instead 04:11:04.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.885124 5356 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/data/instance: 04:11:04.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "2a02b5580c9a4ba983faaaf13e658062" 04:11:04.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:04 on dist-test-slave-dist-test-slave-fvsv" 04:11:04.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.885824 5356 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/wal/instance: 04:11:04.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "2a02b5580c9a4ba983faaaf13e658062" 04:11:04.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:04 on dist-test-slave-dist-test-slave-fvsv" 04:11:04.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.894661 5356 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s 04:11:04.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.895884 5356 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:04.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.896385 5356 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.902925 5356 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.903069 5356 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/wal 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "2a02b5580c9a4ba983faaaf13e658062" 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:04 on dist-test-slave-dist-test-slave-fvsv" 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:04.903308 5356 fs_report.cc:352] FS layout report 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/wal 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/wal 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/data/data 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:04.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:05.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.043139 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } attempt: 4 04:11:05.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:05.044723 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.125:34095: Network error: Client connection negotiation failed: client connection to 127.5.37.125:34095: connect: Connection refused (error 111) 04:11:05.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.348039 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } attempt: 5 04:11:05.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:05.349658 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.125:34095: Network error: Client connection negotiation failed: client connection to 127.5.37.125:34095: connect: Connection refused (error 111) 04:11:05.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:05.761304 5356 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.834s user 0.255s sys 0.578s 04:11:05.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:05.761510 5356 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.835s user 0.255s sys 0.578s 04:11:05.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.805328 5356 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:05.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.874676 5356 rpc_server.cc:205] RPC server started. Bound to: 127.5.37.125:34095 04:11:05.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.877027 5356 webserver.cc:374] Webserver started at http://127.5.37.125:44295/ using document root and password file 04:11:05.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.877892 5356 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-1/data/info.pb 04:11:05.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.878696 5269 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 5356 04:11:05.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.881229 5418 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:05.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.883972 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } attempt: 6 04:11:05.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.5.37.124@KRBTEST.COM; defaulting to no policy 04:11:05.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:05.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.5.37.124@KRBTEST.COM" created. 04:11:05.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.910324 5418 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } has no permanent_uuid. Determining permanent_uuid... 04:11:05.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:05 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.126@KRBTEST.COM for kudu/127.5.37.125@KRBTEST.COM 04:11:05.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:05 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.125@KRBTEST.COM for kudu/127.5.37.126@KRBTEST.COM 04:11:05.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:05.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.124.keytab. 04:11:05.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.124.keytab. 04:11:05.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.124 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.124.keytab. 04:11:05.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.124 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.124.keytab. 04:11:05.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.928483 5269 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:05.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:05.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/wal 04:11:05.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/data 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.37.124:34225 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.37.124 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/logs 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/data/info.pb 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:05.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:05.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:05.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.124.keytab 04:11:05.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.37.124 04:11:05.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:05.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:05.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:05.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:05.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.5.37.126:45983,127.5.37.125:34095,127.5.37.124:34225 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:05.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.935798 5351 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } has no permanent_uuid. Determining permanent_uuid... 04:11:05.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:05.937649 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:05.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.939664 5418 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } has no permanent_uuid. Determining permanent_uuid... 04:11:05.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:05 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.125@KRBTEST.COM for kudu/127.5.37.125@KRBTEST.COM 04:11:05.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.957701 5418 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } has no permanent_uuid. Determining permanent_uuid... 04:11:05.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:05.959218 5418 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:05.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.970008 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 1 04:11:05.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:05.971565 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:05.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:05.978652 5418 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 1 04:11:05.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:05.980274 5418 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:06.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.028626 5418 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 2 04:11:06.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.030344 5418 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:06.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.039839 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 2 04:11:06.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.041414 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:06.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:06.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.081174 5425 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:06.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.081647 5425 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.5.37.124 04:11:06.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.082015 5425 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:06.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.095105 5425 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:06.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.095264 5425 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:06.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.095357 5425 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:06.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.095464 5425 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:06.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.095585 5425 flags.cc:404] Enabled experimental flag: --principal=kudu/127.5.37.124 04:11:06.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.114425 5425 master_main.cc:98] Master server non-default flags: 04:11:06.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/data 04:11:06.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/wal 04:11:06.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:06.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.5.37.126:45983,127.5.37.125:34095,127.5.37.124:34225 04:11:06.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:06.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.37.124:34225 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.124.keytab 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.37.124 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/data/info.pb 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.37.124 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/logs 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:06.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:06.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:06.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:06.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:06.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:06.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:06.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.117698 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 3 04:11:06.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.119299 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:06.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.128928 5425 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 170016us 04:11:06.130 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:06 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608666, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:06.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.131139 5425 init.cc:359] Logged in from keytab as kudu/127.5.37.124@KRBTEST.COM (short username kudu) 04:11:06.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.134806 5425 fs_manager.cc:263] Metadata directory not provided 04:11:06.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.134927 5425 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:06.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.135196 5425 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:06.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.135305 5425 server_base.cc:449] Attempting to create new FS layout instead 04:11:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.138208 5425 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/data/instance: 04:11:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "99c94af8cccf4473a2081d9ffa77158a" 04:11:06.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:06 on dist-test-slave-dist-test-slave-fvsv" 04:11:06.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.138902 5425 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/wal/instance: 04:11:06.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "99c94af8cccf4473a2081d9ffa77158a" 04:11:06.139 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:06 on dist-test-slave-dist-test-slave-fvsv" 04:11:06.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.141693 5418 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 3 04:11:06.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.143357 5418 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:06.148 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.148625 5425 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.005s sys 0.003s 04:11:06.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.149817 5425 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:06.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.150310 5425 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:06.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.157496 5425 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:11:06.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.157616 5425 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/wal 04:11:06.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "99c94af8cccf4473a2081d9ffa77158a" 04:11:06.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:06 on dist-test-slave-dist-test-slave-fvsv" 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.157824 5425 fs_report.cc:352] FS layout report 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/wal 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/wal 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/data/data 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:06.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:06.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.256613 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 4 04:11:06.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.258914 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:06.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.319635 5418 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 4 04:11:06.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.321527 5418 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:06.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.542369 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 5 04:11:06.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.544090 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:06.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:06.624785 5418 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 5 04:11:06.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.626358 5418 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:06.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.982113 5425 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.795s user 0.001s sys 0.000s 04:11:06.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:06.982386 5425 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.795s user 0.001s sys 0.000s 04:11:07.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.012357 5425 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:07.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.056393 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 6 04:11:07.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:07.059587 5351 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.37.124:34225: Network error: Client connection negotiation failed: client connection to 127.5.37.124:34225: connect: Connection refused (error 111) 04:11:07.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.086751 5425 rpc_server.cc:205] RPC server started. Bound to: 127.5.37.124:34225 04:11:07.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.089474 5425 webserver.cc:374] Webserver started at http://127.5.37.124:41955/ using document root and password file 04:11:07.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.090374 5425 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/master-2/data/info.pb 04:11:07.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.093876 5488 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:07.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.094041 5269 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 5425 04:11:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.5.37.65@KRBTEST.COM; defaulting to no policy 04:11:07.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.110597 5488 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } has no permanent_uuid. Determining permanent_uuid... 04:11:07.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:07.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.5.37.65@KRBTEST.COM" created. 04:11:07.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:07 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608666, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.124@KRBTEST.COM for kudu/127.5.37.126@KRBTEST.COM 04:11:07.137 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.136857 5488 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } has no permanent_uuid. Determining permanent_uuid... 04:11:07.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:07.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.65.keytab. 04:11:07.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.65.keytab. 04:11:07.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.65 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.65.keytab. 04:11:07.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.65 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.65.keytab. 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.142168 5269 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-tserver 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-tserver 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/wal 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/data 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.37.65:0 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.5.37.65 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.37.65 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.5.37.126:45983,127.5.37.125:34095,127.5.37.124:34225 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:07.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:07.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/logs 04:11:07.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/data/info.pb 04:11:07.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:07.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:07.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:07.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:07.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.65.keytab 04:11:07.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.37.65 04:11:07.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:07.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:07.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:07.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:07 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608666, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.124@KRBTEST.COM for kudu/127.5.37.125@KRBTEST.COM 04:11:07.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.159555 5488 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } has no permanent_uuid. Determining permanent_uuid... 04:11:07.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.160775 5418 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 6 04:11:07.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:07 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608666, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.124@KRBTEST.COM for kudu/127.5.37.124@KRBTEST.COM 04:11:07.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:07 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608664, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.125@KRBTEST.COM for kudu/127.5.37.124@KRBTEST.COM 04:11:07.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.197808 5488 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a: Bootstrap starting. 04:11:07.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.199162 5418 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062: Bootstrap starting. 04:11:07.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.201009 5488 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a: No blocks or log segments found. Creating new log. 04:11:07.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.202428 5488 log.cc:525] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a: Log is configured to *not* fsync() on all Append() calls 04:11:07.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.202854 5418 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062: No blocks or log segments found. Creating new log. 04:11:07.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.204672 5418 log.cc:525] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062: Log is configured to *not* fsync() on all Append() calls 04:11:07.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.209297 5418 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062: No bootstrap required, opened a new log 04:11:07.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.210126 5488 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a: No bootstrap required, opened a new log 04:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.222033 5488 raft_consensus.cc:340] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } 04:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.222563 5488 raft_consensus.cc:366] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:07.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.222774 5488 raft_consensus.cc:705] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c94af8cccf4473a2081d9ffa77158a, State: Initialized, Role: FOLLOWER 04:11:07.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.223399 5488 consensus_queue.cc:229] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } 04:11:07.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.227164 5418 raft_consensus.cc:340] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } 04:11:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.227849 5418 raft_consensus.cc:366] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.228065 5418 raft_consensus.cc:705] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a02b5580c9a4ba983faaaf13e658062, State: Initialized, Role: FOLLOWER 04:11:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.227735 5488 sys_catalog.cc:339] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:07.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.228334 5488 sys_catalog.cc:342] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [sys.catalog]: This master's current role is: FOLLOWER 04:11:07.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.229290 5488 sys_catalog.cc:427] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [sys.catalog]: configured and running, proceeding with master startup. 04:11:07.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.228770 5418 consensus_queue.cc:229] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } 04:11:07.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.236404 5418 sys_catalog.cc:339] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:07.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.237565 5418 sys_catalog.cc:342] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [sys.catalog]: This master's current role is: FOLLOWER 04:11:07.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.238890 5418 sys_catalog.cc:427] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [sys.catalog]: configured and running, proceeding with master startup. 04:11:07.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.228054 5500 sys_catalog.cc:339] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:07.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.313397 5500 sys_catalog.cc:342] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [sys.catalog]: This master's current role is: FOLLOWER 04:11:07.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.236680 5502 sys_catalog.cc:339] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:07.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.341276 5502 sys_catalog.cc:342] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [sys.catalog]: This master's current role is: FOLLOWER 04:11:07.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.341724 5502 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:11:07.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.342118 5502 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } 04:11:07.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.353050 5502 leader_election.cc:296] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0bb7e3a23e5460ea8e4f95896eaedc8 (127.5.37.126:45983), 99c94af8cccf4473a2081d9ffa77158a (127.5.37.124:34225) 04:11:07.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:07.267562 5443 tablet.cc:1799] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:07.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.382812 5501 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:11:07.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:07.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:07.385761 5494 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:07.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:07.386168 5494 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.5.37.65 04:11:07.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:07.386554 5494 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:07.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:07.312167 5374 tablet.cc:1799] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:07.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:07.401767 5494 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:07.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:07.402024 5494 flags.cc:404] Enabled experimental flag: --principal=kudu/127.5.37.65 04:11:07.402 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:07.402460 5494 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.37.65 04:11:07.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.402715 5464 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a02b5580c9a4ba983faaaf13e658062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c94af8cccf4473a2081d9ffa77158a" is_pre_election: true 04:11:07.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.403688 5464 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a02b5580c9a4ba983faaaf13e658062 in term 0. 04:11:07.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.405408 5365 leader_election.cc:310] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a02b5580c9a4ba983faaaf13e658062, 99c94af8cccf4473a2081d9ffa77158a; no voters: 04:11:07.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.406975 5504 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [term 0 FOLLOWER]: Leader pre-election won for term 1 04:11:07.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.407269 5504 raft_consensus.cc:472] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:11:07.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.407526 5504 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [term 0 FOLLOWER]: Advancing to term 1 04:11:07.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.413234 5504 raft_consensus.cc:494] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } 04:11:07.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.416774 5464 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a02b5580c9a4ba983faaaf13e658062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c94af8cccf4473a2081d9ffa77158a" 04:11:07.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.417332 5464 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [term 0 FOLLOWER]: Advancing to term 1 04:11:07.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.419271 5504 leader_election.cc:296] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [CANDIDATE]: Term 1 election: Requested vote from peers d0bb7e3a23e5460ea8e4f95896eaedc8 (127.5.37.126:45983), 99c94af8cccf4473a2081d9ffa77158a (127.5.37.124:34225) 04:11:07.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:07.422451 5503 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:11:07.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.422593 5464 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a02b5580c9a4ba983faaaf13e658062 in term 1. 04:11:07.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:07.422706 5503 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:11:07.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.423511 5365 leader_election.cc:310] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a02b5580c9a4ba983faaaf13e658062, 99c94af8cccf4473a2081d9ffa77158a; no voters: 04:11:07.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.424232 5504 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [term 1 FOLLOWER]: Leader election won for term 1 04:11:07.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.424852 5504 raft_consensus.cc:667] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [term 1 LEADER]: Becoming Leader. State: Replica: 2a02b5580c9a4ba983faaaf13e658062, State: Running, Role: LEADER 04:11:07.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.425562 5504 consensus_queue.cc:206] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.427820 5494 tablet_server_main.cc:78] Tablet server non-default flags: 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/data 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/wal 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.37.65:0 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.65.keytab 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.37.65 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/data/info.pb 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.37.65 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.5.37.126:45983,127.5.37.125:34095,127.5.37.124:34225 04:11:07.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.5.37.65 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/logs 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:07.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:07.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.430881 5502 sys_catalog.cc:339] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a02b5580c9a4ba983faaaf13e658062. Latest consensus state: current_term: 1 leader_uuid: "2a02b5580c9a4ba983faaaf13e658062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:07.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.431622 5502 sys_catalog.cc:342] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [sys.catalog]: This master's current role is: LEADER 04:11:07.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.437922 5511 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:11:07.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.441076 5511 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:11:07.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.453639 5494 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 170516us 04:11:07.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:07 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608667, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:07.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.456787 5494 init.cc:359] Logged in from keytab as kudu/127.5.37.65@KRBTEST.COM (short username kudu) 04:11:07.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.464063 5494 fs_manager.cc:263] Metadata directory not provided 04:11:07.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.464252 5494 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:07.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.464639 5494 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:07.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.464809 5494 server_base.cc:449] Attempting to create new FS layout instead 04:11:07.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.469231 5494 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/data/instance: 04:11:07.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "186002d601fb4972b2131b629e812ae7" 04:11:07.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:07 on dist-test-slave-dist-test-slave-fvsv" 04:11:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.470270 5494 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/wal/instance: 04:11:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "186002d601fb4972b2131b629e812ae7" 04:11:07.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:07 on dist-test-slave-dist-test-slave-fvsv" 04:11:07.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.480482 5464 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [term 1 FOLLOWER]: Refusing update from remote peer 2a02b5580c9a4ba983faaaf13e658062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:11:07.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.482137 5504 consensus_queue.cc:984] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:11:07.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.484621 5494 fs_manager.cc:503] Time spent creating directory manager: real 0.014s user 0.012s sys 0.004s 04:11:07.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.486256 5494 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:07.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.486950 5494 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:07.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.499450 5494 fs_manager.cc:419] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s 04:11:07.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.499646 5494 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/wal 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "186002d601fb4972b2131b629e812ae7" 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:07 on dist-test-slave-dist-test-slave-fvsv" 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.499977 5494 fs_report.cc:352] FS layout report 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/wal 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/wal 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/data/data 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:07.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:07.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:07.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:07.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:07.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.504881 5500 sys_catalog.cc:339] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a02b5580c9a4ba983faaaf13e658062. Latest consensus state: current_term: 1 leader_uuid: "2a02b5580c9a4ba983faaaf13e658062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:07.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.505615 5500 sys_catalog.cc:342] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [sys.catalog]: This master's current role is: FOLLOWER 04:11:07.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.511375 5504 sys_catalog.cc:339] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a02b5580c9a4ba983faaaf13e658062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:07.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.512734 5504 sys_catalog.cc:342] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [sys.catalog]: This master's current role is: LEADER 04:11:07.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.513681 5516 mvcc.cc:203] Tried to move safe_time back from 6396349101982580736 to 6396349101782310912. Current Snapshot: MvccSnapshot[committed={T|T < 6396349101982580736}] 04:11:07.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.513833 5514 mvcc.cc:203] Tried to move safe_time back from 6396349101982580736 to 6396349101782310912. Current Snapshot: MvccSnapshot[committed={T|T < 6396349101982580736 or (T in {6396349101982580736})}] 04:11:07.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.517287 5511 catalog_manager.cc:959] Generated new certificate authority record 04:11:07.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.519613 5500 sys_catalog.cc:339] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a02b5580c9a4ba983faaaf13e658062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:07.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.520293 5500 sys_catalog.cc:342] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [sys.catalog]: This master's current role is: FOLLOWER 04:11:07.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.521121 5511 catalog_manager.cc:1084] Loading token signing keys... 04:11:07.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:07.554702 5511 catalog_manager.cc:4350] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062: Generated new TSK 0 04:11:08.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.086927 5351 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } attempt: 7 04:11:08.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:08 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608663, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.126@KRBTEST.COM for kudu/127.5.37.124@KRBTEST.COM 04:11:08.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.156090 5351 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8: Bootstrap starting. 04:11:08.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.165940 5351 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8: No blocks or log segments found. Creating new log. 04:11:08.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.170581 5351 log.cc:525] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8: Log is configured to *not* fsync() on all Append() calls 04:11:08.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.179420 5351 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8: No bootstrap required, opened a new log 04:11:08.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.226646 5351 raft_consensus.cc:340] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } 04:11:08.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.229135 5351 raft_consensus.cc:366] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:08.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.229909 5351 raft_consensus.cc:705] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0bb7e3a23e5460ea8e4f95896eaedc8, State: Initialized, Role: FOLLOWER 04:11:08.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.232080 5351 consensus_queue.cc:229] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } 04:11:08.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.237448 5528 sys_catalog.cc:339] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:08.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.238955 5528 sys_catalog.cc:342] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [sys.catalog]: This master's current role is: FOLLOWER 04:11:08.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.392966 5501 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a: acquiring CA information for follower catalog manager: success 04:11:08.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.400176 5501 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:08.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:08.403206 5494 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.863s user 0.289s sys 0.558s 04:11:08.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:08.404114 5494 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.864s user 0.289s sys 0.558s 04:11:08.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.237545 5351 sys_catalog.cc:339] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:08.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.446741 5351 sys_catalog.cc:342] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [sys.catalog]: This master's current role is: FOLLOWER 04:11:08.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.448169 5351 sys_catalog.cc:427] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [sys.catalog]: configured and running, proceeding with master startup. 04:11:08.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.449239 5326 raft_consensus.cc:2890] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [term 0 FOLLOWER]: Advancing to term 1 04:11:08.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.450024 5327 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a02b5580c9a4ba983faaaf13e658062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" 04:11:08.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.448276 5328 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a02b5580c9a4ba983faaaf13e658062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" is_pre_election: true 04:11:08.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.456158 5326 raft_consensus.cc:1184] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [term 1 FOLLOWER]: Refusing update from remote peer 2a02b5580c9a4ba983faaaf13e658062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:11:08.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.456943 5327 raft_consensus.cc:2325] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a02b5580c9a4ba983faaaf13e658062 for term 1 because replica is already servicing an update from a current leader or another vote. 04:11:08.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.458024 5327 rpcz_store.cc:269] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35298 (request call id 3) took 1042ms. Request Metrics: {"spinlock_wait_cycles":14178688} 04:11:08.460 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.459411 5328 raft_consensus.cc:2325] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2a02b5580c9a4ba983faaaf13e658062 for term 1 because replica is already servicing an update from a current leader or another vote. 04:11:08.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.459980 5328 rpcz_store.cc:269] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35298 (request call id 2) took 1108ms. Request Metrics: {"spinlock_wait_cycles":18967936} 04:11:08.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.460204 5528 raft_consensus.cc:472] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:11:08.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.460723 5528 raft_consensus.cc:494] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } 04:11:08.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.462550 5528 leader_election.cc:296] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a02b5580c9a4ba983faaaf13e658062 (127.5.37.125:34095), 99c94af8cccf4473a2081d9ffa77158a (127.5.37.124:34225) 04:11:08.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.463204 5502 consensus_queue.cc:984] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:11:08.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.465837 5464 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c94af8cccf4473a2081d9ffa77158a" is_pre_election: true 04:11:08.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.466466 5464 raft_consensus.cc:2311] T 00000000000000000000000000000000 P 99c94af8cccf4473a2081d9ffa77158a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0bb7e3a23e5460ea8e4f95896eaedc8 for term 2 because replica is either leader or believes a valid leader to be alive. 04:11:08.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.467715 5395 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a02b5580c9a4ba983faaaf13e658062" is_pre_election: true 04:11:08.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.467990 5531 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:11:08.469 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.468412 5395 raft_consensus.cc:2311] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate d0bb7e3a23e5460ea8e4f95896eaedc8 for term 2 because replica is either leader or believes a valid leader to be alive. 04:11:08.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.470558 5298 leader_election.cc:310] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0bb7e3a23e5460ea8e4f95896eaedc8; no voters: 2a02b5580c9a4ba983faaaf13e658062, 99c94af8cccf4473a2081d9ffa77158a 04:11:08.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.506692 5532 raft_consensus.cc:2592] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority 04:11:08.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.507426 5528 sys_catalog.cc:339] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a02b5580c9a4ba983faaaf13e658062. Latest consensus state: current_term: 1 leader_uuid: "2a02b5580c9a4ba983faaaf13e658062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:08.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.508250 5528 sys_catalog.cc:342] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [sys.catalog]: This master's current role is: FOLLOWER 04:11:08.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.507851 5535 sys_catalog.cc:339] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a02b5580c9a4ba983faaaf13e658062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:08.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.508673 5535 sys_catalog.cc:342] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8 [sys.catalog]: This master's current role is: FOLLOWER 04:11:08.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.532171 5494 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:08.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.540706 5494 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) 04:11:08.541 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.541209 5494 ts_tablet_manager.cc:377] Registered 0 tablets 04:11:08.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.868212 5502 sys_catalog.cc:339] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a02b5580c9a4ba983faaaf13e658062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bb7e3a23e5460ea8e4f95896eaedc8" member_type: VOTER last_known_addr { host: "127.5.37.126" port: 45983 } } peers { permanent_uuid: "2a02b5580c9a4ba983faaaf13e658062" member_type: VOTER last_known_addr { host: "127.5.37.125" port: 34095 } } peers { permanent_uuid: "99c94af8cccf4473a2081d9ffa77158a" member_type: VOTER last_known_addr { host: "127.5.37.124" port: 34225 } } } 04:11:08.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:08.870132 5502 sys_catalog.cc:342] T 00000000000000000000000000000000 P 2a02b5580c9a4ba983faaaf13e658062 [sys.catalog]: This master's current role is: LEADER 04:11:09.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.099524 5494 rpc_server.cc:205] RPC server started. Bound to: 127.5.37.65:40005 04:11:09.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.111613 5494 webserver.cc:374] Webserver started at http://127.5.37.65:43163/ using document root and password file 04:11:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.112999 5494 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-0/data/info.pb 04:11:09.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.116757 5269 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-tserver as pid 5494 04:11:09.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.5.37.66@KRBTEST.COM; defaulting to no policy 04:11:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.5.37.66@KRBTEST.COM" created. 04:11:09.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:09 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608667, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.65@KRBTEST.COM for kudu/127.5.37.126@KRBTEST.COM 04:11:09.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:09 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608667, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.65@KRBTEST.COM for kudu/127.5.37.124@KRBTEST.COM 04:11:09.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:09 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608667, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.65@KRBTEST.COM for kudu/127.5.37.125@KRBTEST.COM 04:11:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.66.keytab. 04:11:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.66.keytab. 04:11:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.66 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.66.keytab. 04:11:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.66 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.66.keytab. 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.207088 5269 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-tserver 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-tserver 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/wal 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/data 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.37.66:0 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.5.37.66 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.37.66 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.5.37.126:45983,127.5.37.125:34095,127.5.37.124:34225 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:09.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:09.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:09.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/logs 04:11:09.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/data/info.pb 04:11:09.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:09.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:09.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:09.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:09.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.66.keytab 04:11:09.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.37.66 04:11:09.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:09.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:09.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:09.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.212010 5649 heartbeater.cc:324] Connected to a master server at 127.5.37.124:34225 04:11:09.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.212393 5649 heartbeater.cc:405] Registering TS with master... 04:11:09.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.213140 5649 heartbeater.cc:454] Master 127.5.37.124:34225 requested a full tablet report, sending... 04:11:09.218 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.216764 5454 ts_manager.cc:161] Registered new tserver with Master: 186002d601fb4972b2131b629e812ae7 (127.5.37.65:40005) 04:11:09.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.220962 5656 heartbeater.cc:324] Connected to a master server at 127.5.37.126:45983 04:11:09.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.221268 5656 heartbeater.cc:405] Registering TS with master... 04:11:09.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.221969 5656 heartbeater.cc:454] Master 127.5.37.126:45983 requested a full tablet report, sending... 04:11:09.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.224663 5318 ts_manager.cc:161] Registered new tserver with Master: 186002d601fb4972b2131b629e812ae7 (127.5.37.65:40005) 04:11:09.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.231011 5650 heartbeater.cc:324] Connected to a master server at 127.5.37.125:34095 04:11:09.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.231356 5650 heartbeater.cc:405] Registering TS with master... 04:11:09.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.231848 5650 heartbeater.cc:454] Master 127.5.37.125:34095 requested a full tablet report, sending... 04:11:09.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.233918 5385 ts_manager.cc:161] Registered new tserver with Master: 186002d601fb4972b2131b629e812ae7 (127.5.37.65:40005) 04:11:09.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.237390 5385 master_service.cc:274] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.37.65@KRBTEST.COM'} at 127.5.37.65:54069 04:11:09.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:09.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:09.448616 5661 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:09.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:09.449051 5661 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.5.37.66 04:11:09.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:09.449437 5661 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:09.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:09.466944 5661 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:09.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:09.467263 5661 flags.cc:404] Enabled experimental flag: --principal=kudu/127.5.37.66 04:11:09.467 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:09.467772 5661 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.37.66 04:11:09.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.478791 5531 catalog_manager.cc:1120] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8: acquiring CA information for follower catalog manager: success 04:11:09.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.481688 5531 catalog_manager.cc:1148] T 00000000000000000000000000000000 P d0bb7e3a23e5460ea8e4f95896eaedc8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:09.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.500378 5661 tablet_server_main.cc:78] Tablet server non-default flags: 04:11:09.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/data 04:11:09.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/wal 04:11:09.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.37.66:0 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.66.keytab 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.37.66 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/data/info.pb 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.37.66 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.5.37.126:45983,127.5.37.125:34095,127.5.37.124:34225 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:09.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.5.37.66 04:11:09.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/logs 04:11:09.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:09.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:09.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 04:11:09.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:09.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:09.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:09.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:09.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:09.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:09.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.529613 5661 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 171516us 04:11:09.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:09 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608669, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:09.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.534104 5661 init.cc:359] Logged in from keytab as kudu/127.5.37.66@KRBTEST.COM (short username kudu) 04:11:09.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.543903 5661 fs_manager.cc:263] Metadata directory not provided 04:11:09.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.544101 5661 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:09.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.544461 5661 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:09.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.544632 5661 server_base.cc:449] Attempting to create new FS layout instead 04:11:09.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.549010 5661 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/data/instance: 04:11:09.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1297594e7d7346899595cc70a2c25cd6" 04:11:09.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:09 on dist-test-slave-dist-test-slave-fvsv" 04:11:09.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.550029 5661 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/wal/instance: 04:11:09.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1297594e7d7346899595cc70a2c25cd6" 04:11:09.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:09 on dist-test-slave-dist-test-slave-fvsv" 04:11:09.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.570008 5661 fs_manager.cc:503] Time spent creating directory manager: real 0.019s user 0.013s sys 0.007s 04:11:09.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.572130 5661 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:09.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.573061 5661 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:09.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.584475 5661 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.584656 5661 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/wal 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1297594e7d7346899595cc70a2c25cd6" 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:09 on dist-test-slave-dist-test-slave-fvsv" 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:09.584966 5661 fs_report.cc:352] FS layout report 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/wal 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/wal 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/data/data 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:09.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:09.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:09.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:09.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:10.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.242570 5650 heartbeater.cc:446] Master 127.5.37.125:34095 was elected leader, sending a full tablet report... 04:11:10.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:10.420493 5661 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.788s user 0.275s sys 0.419s 04:11:10.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:10.421030 5661 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.789s user 0.276s sys 0.420s 04:11:10.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.523068 5661 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:10.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.533077 5661 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) 04:11:10.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.533535 5661 ts_tablet_manager.cc:377] Registered 0 tablets 04:11:10.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.863468 5661 rpc_server.cc:205] RPC server started. Bound to: 127.5.37.66:37397 04:11:10.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.867535 5661 webserver.cc:374] Webserver started at http://127.5.37.66:43363/ using document root and password file 04:11:10.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.868340 5661 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-1/data/info.pb 04:11:10.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.875430 5269 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-tserver as pid 5661 04:11:10.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.5.37.67@KRBTEST.COM; defaulting to no policy 04:11:10.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:10.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.5.37.67@KRBTEST.COM" created. 04:11:10.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:10 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608669, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.66@KRBTEST.COM for kudu/127.5.37.125@KRBTEST.COM 04:11:10.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:10 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608669, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.66@KRBTEST.COM for kudu/127.5.37.124@KRBTEST.COM 04:11:10.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:10 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608669, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.66@KRBTEST.COM for kudu/127.5.37.126@KRBTEST.COM 04:11:10.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.937824 5785 heartbeater.cc:324] Connected to a master server at 127.5.37.125:34095 04:11:10.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.938159 5785 heartbeater.cc:405] Registering TS with master... 04:11:10.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.938709 5785 heartbeater.cc:454] Master 127.5.37.125:34095 requested a full tablet report, sending... 04:11:10.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.940426 5385 ts_manager.cc:161] Registered new tserver with Master: 1297594e7d7346899595cc70a2c25cd6 (127.5.37.66:37397) 04:11:10.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.943874 5385 master_service.cc:274] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.37.66@KRBTEST.COM'} at 127.5.37.66:46493 04:11:10.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.945329 5784 heartbeater.cc:324] Connected to a master server at 127.5.37.124:34225 04:11:10.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.945561 5784 heartbeater.cc:405] Registering TS with master... 04:11:10.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.946615 5784 heartbeater.cc:454] Master 127.5.37.124:34225 requested a full tablet report, sending... 04:11:10.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.948459 5788 heartbeater.cc:324] Connected to a master server at 127.5.37.126:45983 04:11:10.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.948691 5788 heartbeater.cc:405] Registering TS with master... 04:11:10.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.948755 5454 ts_manager.cc:161] Registered new tserver with Master: 1297594e7d7346899595cc70a2c25cd6 (127.5.37.66:37397) 04:11:10.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.949076 5788 heartbeater.cc:454] Master 127.5.37.126:45983 requested a full tablet report, sending... 04:11:10.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:10.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.67.keytab. 04:11:10.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.67.keytab. 04:11:10.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.67 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.67.keytab. 04:11:10.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.37.67 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.67.keytab. 04:11:10.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.951447 5318 ts_manager.cc:161] Registered new tserver with Master: 1297594e7d7346899595cc70a2c25cd6 (127.5.37.66:37397) 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:10.951555 5269 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-tserver 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-tserver 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/wal 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/data 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.37.67:0 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.5.37.67 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.37.67 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.5.37.126:45983,127.5.37.125:34095,127.5.37.124:34225 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:10.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/logs 04:11:10.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/data/info.pb 04:11:10.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:10.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:10.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:10.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:10.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.67.keytab 04:11:10.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.37.67 04:11:10.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:10.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:10.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:11.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:11.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:11.095963 5796 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:11.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:11.096282 5796 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.5.37.67 04:11:11.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:11.096515 5796 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:11.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:11.107153 5796 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:11.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:11.107354 5796 flags.cc:404] Enabled experimental flag: --principal=kudu/127.5.37.67 04:11:11.107 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:11.107656 5796 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.37.67 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.126936 5796 tablet_server_main.cc:78] Tablet server non-default flags: 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/data 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/wal 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.37.67:0 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kudu_127.5.37.67.keytab 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.37.67 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:11.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/data/info.pb 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.37.67 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.5.37.126:45983,127.5.37.125:34095,127.5.37.124:34225 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.5.37.67 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/logs 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:11.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:11.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.144549 5796 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 172516us 04:11:11.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:11 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608671, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:11.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.146917 5796 init.cc:359] Logged in from keytab as kudu/127.5.37.67@KRBTEST.COM (short username kudu) 04:11:11.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.151582 5796 fs_manager.cc:263] Metadata directory not provided 04:11:11.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.151731 5796 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:11.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.152014 5796 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:11.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.152134 5796 server_base.cc:449] Attempting to create new FS layout instead 04:11:11.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.155253 5796 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/data/instance: 04:11:11.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "c43cf95b6e2a447d81fc970c99a84d3c" 04:11:11.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:11 on dist-test-slave-dist-test-slave-fvsv" 04:11:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.156038 5796 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/wal/instance: 04:11:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "c43cf95b6e2a447d81fc970c99a84d3c" 04:11:11.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:11 on dist-test-slave-dist-test-slave-fvsv" 04:11:11.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.164901 5796 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.008s sys 0.001s 04:11:11.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.166064 5796 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:11.166 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.166601 5796 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:11.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.184489 5796 fs_manager.cc:419] Time spent opening block manager: real 0.013s user 0.004s sys 0.008s 04:11:11.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.184641 5796 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/wal 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "c43cf95b6e2a447d81fc970c99a84d3c" 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:11 on dist-test-slave-dist-test-slave-fvsv" 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.184957 5796 fs_report.cc:352] FS layout report 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/wal 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/wal 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/data/data 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:11.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:11.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:11.825873 5796 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.607s user 0.001s sys 0.001s 04:11:11.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:11.826092 5796 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.607s user 0.001s sys 0.001s 04:11:11.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.872470 5796 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:11.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.877449 5796 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) 04:11:11.877 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.877647 5796 ts_tablet_manager.cc:377] Registered 0 tablets 04:11:11.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:11.947268 5785 heartbeater.cc:446] Master 127.5.37.125:34095 was elected leader, sending a full tablet report... 04:11:12.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.053057 5796 rpc_server.cc:205] RPC server started. Bound to: 127.5.37.67:34315 04:11:12.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.056497 5796 webserver.cc:374] Webserver started at http://127.5.37.67:42669/ using document root and password file 04:11:12.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.057320 5796 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/ts-2/data/info.pb 04:11:12.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.063565 5269 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-tserver as pid 5796 04:11:12.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:12 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.37.124@KRBTEST.COM 04:11:12.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:12 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608671, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.67@KRBTEST.COM for kudu/127.5.37.124@KRBTEST.COM 04:11:12.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:12 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.37.125@KRBTEST.COM 04:11:12.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:12 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608671, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.67@KRBTEST.COM for kudu/127.5.37.125@KRBTEST.COM 04:11:12.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:12 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608671, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.37.67@KRBTEST.COM for kudu/127.5.37.126@KRBTEST.COM 04:11:12.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.141127 5920 heartbeater.cc:324] Connected to a master server at 127.5.37.125:34095 04:11:12.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.141486 5920 heartbeater.cc:405] Registering TS with master... 04:11:12.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.142163 5920 heartbeater.cc:454] Master 127.5.37.125:34095 requested a full tablet report, sending... 04:11:12.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.143985 5385 ts_manager.cc:161] Registered new tserver with Master: c43cf95b6e2a447d81fc970c99a84d3c (127.5.37.67:34315) 04:11:12.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.145635 5385 master_service.cc:274] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.37.67@KRBTEST.COM'} at 127.5.37.67:56673 04:11:12.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.147619 5919 heartbeater.cc:324] Connected to a master server at 127.5.37.124:34225 04:11:12.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.147858 5919 heartbeater.cc:405] Registering TS with master... 04:11:12.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.148819 5919 heartbeater.cc:454] Master 127.5.37.124:34225 requested a full tablet report, sending... 04:11:12.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.151505 5454 ts_manager.cc:161] Registered new tserver with Master: c43cf95b6e2a447d81fc970c99a84d3c (127.5.37.67:34315) 04:11:12.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.148254 5921 heartbeater.cc:324] Connected to a master server at 127.5.37.126:45983 04:11:12.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.159832 5921 heartbeater.cc:405] Registering TS with master... 04:11:12.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.160178 5921 heartbeater.cc:454] Master 127.5.37.126:45983 requested a full tablet report, sending... 04:11:12.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.161911 5318 ts_manager.cc:161] Registered new tserver with Master: c43cf95b6e2a447d81fc970c99a84d3c (127.5.37.67:34315) 04:11:12.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:12 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.37.126@KRBTEST.COM 04:11:12.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.178691 5269 external_mini_cluster.cc:628] 3 TS(s) registered with all masters 04:11:12.178 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:11:12.179 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_kdc_env_vars { } 04:11:12.179 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_kdc_env_vars { env_vars { key: "KRB5_CONFIG" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5.conf" } env_vars { key: "KRB5CCNAME" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5cc" } env_vars { key: "KUDU_ENABLE_KRB5_REALM_FIX" value: "yes" } env_vars { key: "KRB5_KDC_PROFILE" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/kdc.conf" } } 04:11:12.180 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_masters { } 04:11:12.180 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.5.37.126" port: 45983 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.5.37.125" port: 34095 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.5.37.124" port: 34225 } } } 04:11:12.182 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_tservers { } 04:11:12.183 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_tservers { tservers { id { type: TSERVER index: 0 } bound_rpc_address { host: "127.5.37.65" port: 40005 } } tservers { id { type: TSERVER index: 1 } bound_rpc_address { host: "127.5.37.66" port: 37397 } } tservers { id { type: TSERVER index: 2 } bound_rpc_address { host: "127.5.37.67" port: 34315 } } } 04:11:12.184 [INFO - main] (MiniKuduCluster.java:439) Running kinit for user test-admin 04:11:12.184 [DEBUG - main] (MiniKuduCluster.java:155) Request: kinit { username: "test-admin" } 04:11:12.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:12 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:12.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:11:12.209 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:11:12.214 [DEBUG - main] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6900904995252879440/krb5kdc/krb5cc 04:11:12.215 [DEBUG - main] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:11:12.220 [DEBUG - New I/O worker #74] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.5.37.126 04:11:12.220 [DEBUG - New I/O worker #73] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.5.37.125 04:11:12.221 [DEBUG - New I/O worker #75] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.5.37.124 04:11:12.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:12 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.37.126@KRBTEST.COM 04:11:12.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:12 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.37.124@KRBTEST.COM 04:11:12.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:12 dist-test-slave-dist-test-slave-fvsv krb5kdc[5278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.37.125@KRBTEST.COM 04:11:12.251 [DEBUG - New I/O worker #74] (Negotiator.java:764) Authenticated connection [id: 0xde2df06b, /127.0.0.1:35346 => /127.5.37.126:45983] using SASL/GSSAPI 04:11:12.255 [DEBUG - New I/O worker #75] (Negotiator.java:764) Authenticated connection [id: 0x71ec0862, /127.0.0.1:60986 => /127.5.37.124:34225] using SASL/GSSAPI 04:11:12.257 [DEBUG - New I/O worker #73] (Negotiator.java:764) Authenticated connection [id: 0xa5e57243, /127.0.0.1:35668 => /127.5.37.125:34095] using SASL/GSSAPI 04:11:12.263 [DEBUG - New I/O worker #73] (AsyncKuduClient.java:2322) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 04:11:12.291 [DEBUG - main] (SecurityUtil.java:121) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 04:11:12.294 [DEBUG - New I/O worker #83] (Connection.java:691) [peer master-127.5.37.126:45983(127.5.37.126:45983)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available 04:11:12.294 [DEBUG - New I/O worker #82] (Connection.java:691) [peer master-127.5.37.125:34095(127.5.37.125:34095)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available 04:11:12.295 [INFO - New I/O worker #83] (ConnectToCluster.java:361) Unable to connect to master 127.5.37.126:45983: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available 04:11:12.295 [INFO - New I/O worker #82] (ConnectToCluster.java:361) Unable to connect to master 127.5.37.125:34095: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available 04:11:12.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:12.296403 5927 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:12.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.293090 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:35352 04:11:12.296 [DEBUG - New I/O worker #84] (Connection.java:691) [peer master-127.5.37.124:34225(127.5.37.124:34225)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available 04:11:12.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.293560 (+ 470us) server_negotiation.cc:184] Beginning negotiation 04:11:12.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.293581 (+ 21us) server_negotiation.cc:373] Waiting for connection header 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.293628 (+ 47us) server_negotiation.cc:381] Connection header received 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.293894 (+ 266us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:12.296 [INFO - New I/O worker #84] (ConnectToCluster.java:361) Unable to connect to master 127.5.37.124:34225: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.293918 (+ 24us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.294045 (+ 127us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.294180 (+ 135us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.295565 (+ 1385us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35352: BlockingRecv error: recv got EOF from 127.0.0.1:35352 (error 108) 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":176} 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:12.296799 5931 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.292694 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:35674 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.294048 (+ 1354us) server_negotiation.cc:184] Beginning negotiation 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.294071 (+ 23us) server_negotiation.cc:373] Waiting for connection header 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.294107 (+ 36us) server_negotiation.cc:381] Connection header received 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.294218 (+ 111us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.294248 (+ 30us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.294366 (+ 118us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.294520 (+ 154us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.295949 (+ 1429us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35674: BlockingRecv error: recv got EOF from 127.0.0.1:35674 (error 108) 04:11:12.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":1068} 04:11:12.297 [DEBUG - New I/O worker #84] (AsyncKuduClient.java:1331) Notify RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=5), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.5.37.125:34095, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.5.37.126:45983, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.5.37.124:34225, [3ms] Sub RPC ConnectToMaster: received response from server master-127.5.37.126:45983: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4ms] Sub RPC ConnectToMaster: received response from server master-127.5.37.125:34095: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [5ms] Sub RPC ConnectToMaster: received response from server master-127.5.37.124:34225: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, deferred=null) after lookup exception org.apache.kudu.client.NonRecoverableException: Couldn't find a valid master in (127.5.37.125:34095,127.5.37.126:45983,127.5.37.124:34225). Exceptions received: [org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available] at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:244) [main/:?] at org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49) [main/:?] at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:363) [main/:?] at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:352) [main/:?] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) [async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) [async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) [async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) [async-1.4.1.jar:?] at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) [async-1.4.1.jar:?] at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275) [main/:?] at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:329) [main/:?] at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:247) [main/:?] at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59) [main/:?] at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149) [main/:?] at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145) [main/:?] at org.apache.kudu.client.Connection.cleanup(Connection.java:719) [main/:?] at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:464) [main/:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [netty-3.10.6.Final.jar:?] at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238) [main/:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [netty-3.10.6.Final.jar:?] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.6.Final.jar:?] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.6.Final.jar:?] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:?] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141] 04:11:12.298 [DEBUG - New I/O worker #84] (KuduRpc.java:258) Handling a callback on RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=6), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.5.37.125:34095, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.5.37.126:45983, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.5.37.124:34225, [3ms] Sub RPC ConnectToMaster: received response from server master-127.5.37.126:45983: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [4ms] Sub RPC ConnectToMaster: received response from server master-127.5.37.125:34095: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [5ms] Sub RPC ConnectToMaster: received response from server master-127.5.37.124:34225: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, deferred=null) with no deferred attached! 04:11:12.299 [DEBUG - main] (SecurityContext.java:312) Importing authentication credentials with one authn token, 1 cert(s), and realUser=slave 04:11:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:12.299880 5922 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.293550 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:60992 04:11:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.293973 (+ 423us) server_negotiation.cc:184] Beginning negotiation 04:11:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.293996 (+ 23us) server_negotiation.cc:373] Waiting for connection header 04:11:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.295686 (+ 1690us) server_negotiation.cc:381] Connection header received 04:11:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.296041 (+ 355us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.296067 (+ 26us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.296192 (+ 125us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.296317 (+ 125us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:12.299002 (+ 2685us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60992: BlockingRecv error: recv got EOF from 127.0.0.1:60992 (error 108) 04:11:12.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":146} 04:11:12.332 [DEBUG - New I/O worker #86] (Negotiator.java:764) Authenticated connection [id: 0x3bb6be05, /127.0.0.1:35358 => /127.5.37.126:45983] using TOKEN/null 04:11:12.333 [DEBUG - New I/O worker #87] (Negotiator.java:764) Authenticated connection [id: 0xaea87e5c, /127.0.0.1:60998 => /127.5.37.124:34225] using TOKEN/null 04:11:12.334 [DEBUG - New I/O worker #85] (Negotiator.java:764) Authenticated connection [id: 0xdf62270a, /127.0.0.1:35680 => /127.5.37.125:34095] using TOKEN/null 04:11:12.343 [DEBUG - New I/O worker #85] (AsyncKuduClient.java:2322) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 04:11:12.344 [DEBUG - New I/O worker #85] (AsyncKuduClient.java:1296) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=25), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.5.37.125:34095, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.5.37.126:45983, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.5.37.124:34225, [17ms] Sub RPC ConnectToMaster: received response from server master-127.5.37.126:45983: OK, [17ms] Sub RPC ConnectToMaster: received response from server master-127.5.37.124:34225: OK, [24ms] Sub RPC ConnectToMaster: received response from server master-127.5.37.125:34095: OK, deferred=null) after lookup 04:11:12.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.382813 5385 catalog_manager.cc:1418] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35680: 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "TestSecurity-table" 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 04:11:12.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 04:11:12.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 04:11:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 04:11:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 04:11:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 04:11:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 04:11:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 04:11:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 04:11:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 04:11:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 04:11:12.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:12.395097 5385 catalog_manager.cc:1583] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestSecurity-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 04:11:12.448 [DEBUG - New I/O worker #85] (AsyncKuduClient.java:776) Opened table 11d14fa2815f470187f43e16b39f1138 04:11:12.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.452827 5583 tablet_service.cc:1016] Processing CreateTablet for tablet f9c2b36ac15c433a97e3d3f320087d2e (table=TestSecurity-table [id=11d14fa2815f470187f43e16b39f1138]), partition=RANGE (key) PARTITION UNBOUNDED 04:11:12.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.453768 5583 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet f9c2b36ac15c433a97e3d3f320087d2e. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:12.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.454133 5856 tablet_service.cc:1016] Processing CreateTablet for tablet f9c2b36ac15c433a97e3d3f320087d2e (table=TestSecurity-table [id=11d14fa2815f470187f43e16b39f1138]), partition=RANGE (key) PARTITION UNBOUNDED 04:11:12.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.454790 5856 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet f9c2b36ac15c433a97e3d3f320087d2e. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:12.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.462810 5721 tablet_service.cc:1016] Processing CreateTablet for tablet f9c2b36ac15c433a97e3d3f320087d2e (table=TestSecurity-table [id=11d14fa2815f470187f43e16b39f1138]), partition=RANGE (key) PARTITION UNBOUNDED 04:11:12.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.464108 5721 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet f9c2b36ac15c433a97e3d3f320087d2e. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:12.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.477579 5970 tablet_bootstrap.cc:439] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c: Bootstrap starting. 04:11:12.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.484098 5971 tablet_bootstrap.cc:439] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7: Bootstrap starting. 04:11:12.485 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.485172 5970 tablet_bootstrap.cc:590] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c: No blocks or log segments found. Creating new log. 04:11:12.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.488739 5970 log.cc:525] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c: Log is configured to *not* fsync() on all Append() calls 04:11:12.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.493970 5971 tablet_bootstrap.cc:590] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7: No blocks or log segments found. Creating new log. 04:11:12.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.497622 5972 tablet_bootstrap.cc:439] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6: Bootstrap starting. 04:11:12.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.497836 5971 log.cc:525] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7: Log is configured to *not* fsync() on all Append() calls 04:11:12.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.501178 5970 tablet_bootstrap.cc:439] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c: No bootstrap required, opened a new log 04:11:12.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.501502 5971 tablet_bootstrap.cc:439] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7: No bootstrap required, opened a new log 04:11:12.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.501863 5970 ts_tablet_manager.cc:1057] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.009s 04:11:12.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.501986 5971 ts_tablet_manager.cc:1057] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s 04:11:12.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.502130 5972 tablet_bootstrap.cc:590] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6: No blocks or log segments found. Creating new log. 04:11:12.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.504966 5972 log.cc:525] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6: Log is configured to *not* fsync() on all Append() calls 04:11:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.510061 5972 tablet_bootstrap.cc:439] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6: No bootstrap required, opened a new log 04:11:12.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.510458 5972 ts_tablet_manager.cc:1057] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s 04:11:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.526310 5971 raft_consensus.cc:340] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1297594e7d7346899595cc70a2c25cd6" member_type: VOTER last_known_addr { host: "127.5.37.66" port: 37397 } } peers { permanent_uuid: "186002d601fb4972b2131b629e812ae7" member_type: VOTER last_known_addr { host: "127.5.37.65" port: 40005 } } peers { permanent_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" member_type: VOTER last_known_addr { host: "127.5.37.67" port: 34315 } } 04:11:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.529255 5971 raft_consensus.cc:366] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:12.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.529469 5971 raft_consensus.cc:705] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186002d601fb4972b2131b629e812ae7, State: Initialized, Role: FOLLOWER 04:11:12.530 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.530082 5971 consensus_queue.cc:229] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1297594e7d7346899595cc70a2c25cd6" member_type: VOTER last_known_addr { host: "127.5.37.66" port: 37397 } } peers { permanent_uuid: "186002d601fb4972b2131b629e812ae7" member_type: VOTER last_known_addr { host: "127.5.37.65" port: 40005 } } peers { permanent_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" member_type: VOTER last_known_addr { host: "127.5.37.67" port: 34315 } } 04:11:12.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.535696 5971 ts_tablet_manager.cc:1085] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7: Time spent starting tablet: real 0.034s user 0.018s sys 0.010s 04:11:12.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.543174 5970 raft_consensus.cc:340] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1297594e7d7346899595cc70a2c25cd6" member_type: VOTER last_known_addr { host: "127.5.37.66" port: 37397 } } peers { permanent_uuid: "186002d601fb4972b2131b629e812ae7" member_type: VOTER last_known_addr { host: "127.5.37.65" port: 40005 } } peers { permanent_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" member_type: VOTER last_known_addr { host: "127.5.37.67" port: 34315 } } 04:11:12.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.543792 5970 raft_consensus.cc:366] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:12.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.543998 5970 raft_consensus.cc:705] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43cf95b6e2a447d81fc970c99a84d3c, State: Initialized, Role: FOLLOWER 04:11:12.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.544584 5970 consensus_queue.cc:229] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1297594e7d7346899595cc70a2c25cd6" member_type: VOTER last_known_addr { host: "127.5.37.66" port: 37397 } } peers { permanent_uuid: "186002d601fb4972b2131b629e812ae7" member_type: VOTER last_known_addr { host: "127.5.37.65" port: 40005 } } peers { permanent_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" member_type: VOTER last_known_addr { host: "127.5.37.67" port: 34315 } } 04:11:12.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.545600 5972 raft_consensus.cc:340] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1297594e7d7346899595cc70a2c25cd6" member_type: VOTER last_known_addr { host: "127.5.37.66" port: 37397 } } peers { permanent_uuid: "186002d601fb4972b2131b629e812ae7" member_type: VOTER last_known_addr { host: "127.5.37.65" port: 40005 } } peers { permanent_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" member_type: VOTER last_known_addr { host: "127.5.37.67" port: 34315 } } 04:11:12.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.546419 5972 raft_consensus.cc:366] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:12.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.546630 5972 raft_consensus.cc:705] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1297594e7d7346899595cc70a2c25cd6, State: Initialized, Role: FOLLOWER 04:11:12.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.547405 5972 consensus_queue.cc:229] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1297594e7d7346899595cc70a2c25cd6" member_type: VOTER last_known_addr { host: "127.5.37.66" port: 37397 } } peers { permanent_uuid: "186002d601fb4972b2131b629e812ae7" member_type: VOTER last_known_addr { host: "127.5.37.65" port: 40005 } } peers { permanent_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" member_type: VOTER last_known_addr { host: "127.5.37.67" port: 34315 } } 04:11:12.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.562412 5972 ts_tablet_manager.cc:1085] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6: Time spent starting tablet: real 0.052s user 0.025s sys 0.004s 04:11:12.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.569404 5920 heartbeater.cc:446] Master 127.5.37.125:34095 was elected leader, sending a full tablet report... 04:11:12.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.576707 5970 ts_tablet_manager.cc:1085] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c: Time spent starting tablet: real 0.074s user 0.018s sys 0.025s 04:11:12.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.577373 5979 raft_consensus.cc:472] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:11:12.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.579228 5979 raft_consensus.cc:494] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1297594e7d7346899595cc70a2c25cd6" member_type: VOTER last_known_addr { host: "127.5.37.66" port: 37397 } } peers { permanent_uuid: "186002d601fb4972b2131b629e812ae7" member_type: VOTER last_known_addr { host: "127.5.37.65" port: 40005 } } peers { permanent_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" member_type: VOTER last_known_addr { host: "127.5.37.67" port: 34315 } } 04:11:12.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.580775 5979 leader_election.cc:296] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1297594e7d7346899595cc70a2c25cd6 (127.5.37.66:37397), 186002d601fb4972b2131b629e812ae7 (127.5.37.65:40005) 04:11:12.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.590355 5603 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "f9c2b36ac15c433a97e3d3f320087d2e" candidate_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186002d601fb4972b2131b629e812ae7" is_pre_election: true 04:11:12.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.590903 5741 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "f9c2b36ac15c433a97e3d3f320087d2e" candidate_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1297594e7d7346899595cc70a2c25cd6" is_pre_election: true 04:11:12.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.591094 5603 raft_consensus.cc:2350] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c43cf95b6e2a447d81fc970c99a84d3c in term 0. 04:11:12.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.591547 5741 raft_consensus.cc:2350] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c43cf95b6e2a447d81fc970c99a84d3c in term 0. 04:11:12.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.592912 5806 leader_election.cc:310] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 186002d601fb4972b2131b629e812ae7, c43cf95b6e2a447d81fc970c99a84d3c; no voters: 04:11:12.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.593655 5979 raft_consensus.cc:2644] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [term 0 FOLLOWER]: Leader pre-election won for term 1 04:11:12.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.594019 5979 raft_consensus.cc:472] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:11:12.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.594200 5979 raft_consensus.cc:2890] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [term 0 FOLLOWER]: Advancing to term 1 04:11:12.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:12.595098 5926 tablet.cc:1799] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:12.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.597862 5979 raft_consensus.cc:494] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1297594e7d7346899595cc70a2c25cd6" member_type: VOTER last_known_addr { host: "127.5.37.66" port: 37397 } } peers { permanent_uuid: "186002d601fb4972b2131b629e812ae7" member_type: VOTER last_known_addr { host: "127.5.37.65" port: 40005 } } peers { permanent_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" member_type: VOTER last_known_addr { host: "127.5.37.67" port: 34315 } } 04:11:12.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.599620 5979 leader_election.cc:296] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [CANDIDATE]: Term 1 election: Requested vote from peers 1297594e7d7346899595cc70a2c25cd6 (127.5.37.66:37397), 186002d601fb4972b2131b629e812ae7 (127.5.37.65:40005) 04:11:12.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.600301 5741 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "f9c2b36ac15c433a97e3d3f320087d2e" candidate_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1297594e7d7346899595cc70a2c25cd6" 04:11:12.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.600643 5741 raft_consensus.cc:2890] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6 [term 0 FOLLOWER]: Advancing to term 1 04:11:12.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.600775 5603 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "f9c2b36ac15c433a97e3d3f320087d2e" candidate_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "186002d601fb4972b2131b629e812ae7" 04:11:12.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.601215 5603 raft_consensus.cc:2890] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7 [term 0 FOLLOWER]: Advancing to term 1 04:11:12.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.605960 5741 raft_consensus.cc:2350] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c43cf95b6e2a447d81fc970c99a84d3c in term 1. 04:11:12.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.606763 5603 raft_consensus.cc:2350] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c43cf95b6e2a447d81fc970c99a84d3c in term 1. 04:11:12.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.606717 5806 leader_election.cc:310] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1297594e7d7346899595cc70a2c25cd6, c43cf95b6e2a447d81fc970c99a84d3c; no voters: 04:11:12.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.607383 5979 raft_consensus.cc:2644] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [term 1 FOLLOWER]: Leader election won for term 1 04:11:12.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.607707 5979 raft_consensus.cc:667] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [term 1 LEADER]: Becoming Leader. State: Replica: c43cf95b6e2a447d81fc970c99a84d3c, State: Running, Role: LEADER 04:11:12.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.608286 5979 consensus_queue.cc:206] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1297594e7d7346899595cc70a2c25cd6" member_type: VOTER last_known_addr { host: "127.5.37.66" port: 37397 } } peers { permanent_uuid: "186002d601fb4972b2131b629e812ae7" member_type: VOTER last_known_addr { host: "127.5.37.65" port: 40005 } } peers { permanent_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" member_type: VOTER last_known_addr { host: "127.5.37.67" port: 34315 } } 04:11:12.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.615417 5385 catalog_manager.cc:4046] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c reported cstate change: term changed from 0 to 1, leader changed from to c43cf95b6e2a447d81fc970c99a84d3c (127.5.37.67). New cstate: current_term: 1 leader_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1297594e7d7346899595cc70a2c25cd6" member_type: VOTER last_known_addr { host: "127.5.37.66" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "186002d601fb4972b2131b629e812ae7" member_type: VOTER last_known_addr { host: "127.5.37.65" port: 40005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c43cf95b6e2a447d81fc970c99a84d3c" member_type: VOTER last_known_addr { host: "127.5.37.67" port: 34315 } health_report { overall_health: HEALTHY } } } 04:11:12.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:12.643703 5792 tablet.cc:1799] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:12.709 [DEBUG - New I/O worker #85] (AsyncKuduClient.java:2322) Learned about tablet f9c2b36ac15c433a97e3d3f320087d2e for table 'TestSecurity-table' with partition [, ) 04:11:12.709 [DEBUG - New I/O worker #85] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299993, tablet=f9c2b36ac15c433a97e3d3f320087d2e@[1297594e7d7346899595cc70a2c25cd6(127.5.37.66:37397),186002d601fb4972b2131b629e812ae7(127.5.37.65:40005),c43cf95b6e2a447d81fc970c99a84d3c(127.5.37.67:34315)[L]]}] 04:11:12.717 [DEBUG - New I/O worker #85] (AsyncKuduClient.java:1296) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=15), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.5.37.125:34095, [7ms] Sub RPC GetTableLocations: received response from server master-127.5.37.125:34095: OK, deferred=null) after lookup 04:11:12.737 [DEBUG - New I/O worker #88] (Negotiator.java:764) Authenticated connection [id: 0x57e5122c, /127.0.0.1:36248 => /127.5.37.67:34315] using TOKEN/null 04:11:12.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.756664 5741 raft_consensus.cc:1184] T f9c2b36ac15c433a97e3d3f320087d2e P 1297594e7d7346899595cc70a2c25cd6 [term 1 FOLLOWER]: Refusing update from remote peer c43cf95b6e2a447d81fc970c99a84d3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:11:12.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.757338 5603 raft_consensus.cc:1184] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7 [term 1 FOLLOWER]: Refusing update from remote peer c43cf95b6e2a447d81fc970c99a84d3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:11:12.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.757905 5979 consensus_queue.cc:984] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1297594e7d7346899595cc70a2c25cd6" member_type: VOTER last_known_addr { host: "127.5.37.66" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:11:12.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.759728 5977 consensus_queue.cc:984] T f9c2b36ac15c433a97e3d3f320087d2e P c43cf95b6e2a447d81fc970c99a84d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "186002d601fb4972b2131b629e812ae7" member_type: VOTER last_known_addr { host: "127.5.37.65" port: 40005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s 04:11:12.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:12.659309 5657 tablet.cc:1799] T f9c2b36ac15c433a97e3d3f320087d2e P 186002d601fb4972b2131b629e812ae7: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:12.816 [DEBUG - New I/O worker #87] (Connection.java:691) [peer master-127.5.37.124:34225(127.5.37.124:34225)] cleaning up while in state READY due to: connection disconnected 04:11:12.816 [DEBUG - New I/O worker #85] (Connection.java:691) [peer master-127.5.37.125:34095(127.5.37.125:34095)] cleaning up while in state READY due to: connection disconnected 04:11:12.819 [DEBUG - New I/O worker #88] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:11:12.819 [DEBUG - New I/O worker #86] (Connection.java:691) [peer master-127.5.37.126:45983(127.5.37.126:45983)] cleaning up while in state READY due to: connection disconnected 04:11:12.819 [DEBUG - New I/O worker #88] (Connection.java:691) [peer c43cf95b6e2a447d81fc970c99a84d3c(127.5.37.67:34315)] cleaning up while in state READY due to: connection disconnected 04:11:12.823 [DEBUG - New I/O worker #75] (Connection.java:691) [peer master-127.5.37.124:34225(127.5.37.124:34225)] cleaning up while in state READY due to: connection disconnected 04:11:12.823 [DEBUG - New I/O worker #73] (Connection.java:691) [peer master-127.5.37.125:34095(127.5.37.125:34095)] cleaning up while in state READY due to: connection disconnected 04:11:12.824 [DEBUG - New I/O worker #74] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:11:12.829 [DEBUG - New I/O worker #74] (Connection.java:691) [peer master-127.5.37.126:45983(127.5.37.126:45983)] cleaning up while in state READY due to: connection disconnected 04:11:12.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.830344 5269 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-tserver with pid 5494 04:11:12.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.851685 5269 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-tserver with pid 5661 04:11:12.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.870712 5269 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-tserver with pid 5796 04:11:12.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.901681 5269 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 5289 04:11:12.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.923612 5269 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 5356 04:11:12.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:12.947593 5269 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 5425 04:11:13.990 [INFO - main] (RetryRule.java:136) Creating RetryStatement with result reporter and retry count of 0 (no retries) .04:11:13.991 [INFO - Time-limited test] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskusvpEt/test-tmp 04:11:13.991 [INFO - Time-limited test] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin 04:11:13.991 [INFO - Time-limited test] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskusvpEt/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb] 04:11:13.993 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 0 enable_kerberos: true cluster_root: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219" hms_mode: NONE mini_kdc_options { } } 04:11:14.169 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:14.170 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_cluster { } 04:11:14.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Loading random data 04:11:14.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Initializing database '/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/principal' for realm 'KRBTEST.COM', 04:11:14.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) master key name 'K/M@KRBTEST.COM' 04:11:14.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:14 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): setting up network... 04:11:14.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:14 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) 04:11:14.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked 04:11:14.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 04:11:14.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:14 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): skipping unrecognized local address family 17 04:11:14.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:14 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): skipping unrecognized local address family 17 04:11:14.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:14 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): set up 1 sockets 04:11:14.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:14 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): commencing operation 04:11:14.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: starting... 04:11:14.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:14.755831 5996 mini_kdc.cc:119] Time spent starting KDC: real 0.560s user 0.007s sys 0.010s 04:11:14.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy 04:11:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-admin@KRBTEST.COM" created. 04:11:14.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy 04:11:14.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:14.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-user@KRBTEST.COM" created. 04:11:14.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy 04:11:14.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:14.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "joe-interloper@KRBTEST.COM" created. 04:11:14.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:14.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/test-user.keytab. 04:11:14.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/test-user.keytab. 04:11:14.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/test-user.keytab. 04:11:14.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/test-user.keytab. 04:11:14.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:14 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:14.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:11:14.885 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.5.219.62@KRBTEST.COM; defaulting to no policy 04:11:14.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:14.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.5.219.62@KRBTEST.COM" created. 04:11:14.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:14.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.219.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.62.keytab. 04:11:14.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.219.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.62.keytab. 04:11:14.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.219.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.62.keytab. 04:11:14.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.219.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.62.keytab. 04:11:14.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:14.910627 5996 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:14.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:14.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/wal 04:11:14.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/data 04:11:14.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.219.62:40075 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.219.62 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/logs 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/data/info.pb 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.62.keytab 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.219.62 04:11:14.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:14.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:14.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:14.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:14.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.5.219.62:40075,127.5.219.61:37941,127.5.219.60:34321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:15.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:15.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:15.057690 6016 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:15.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:15.058049 6016 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.5.219.62 04:11:15.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:15.058286 6016 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:15.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:15.069278 6016 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:15.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:15.069435 6016 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:15.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:15.069530 6016 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:15.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:15.069654 6016 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:15.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:15.069789 6016 flags.cc:404] Enabled experimental flag: --principal=kudu/127.5.219.62 04:11:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.089403 6016 master_main.cc:98] Master server non-default flags: 04:11:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/data 04:11:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/wal 04:11:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.5.219.62:40075,127.5.219.61:37941,127.5.219.60:34321 04:11:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:15.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.219.62:40075 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.62.keytab 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.219.62 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/data/info.pb 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.219.62 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/logs 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:15.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:15.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.103435 6016 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 174516us 04:11:15.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:15 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.219.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:15.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.105906 6016 init.cc:359] Logged in from keytab as kudu/127.5.219.62@KRBTEST.COM (short username kudu) 04:11:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.109254 6016 fs_manager.cc:263] Metadata directory not provided 04:11:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.109400 6016 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.109735 6016 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.109865 6016 server_base.cc:449] Attempting to create new FS layout instead 04:11:15.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.112924 6016 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/data/instance: 04:11:15.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "3295f134662d4295a59d31c8ad94a3e2" 04:11:15.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:15 on dist-test-slave-dist-test-slave-fvsv" 04:11:15.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.113752 6016 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/wal/instance: 04:11:15.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "3295f134662d4295a59d31c8ad94a3e2" 04:11:15.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:15 on dist-test-slave-dist-test-slave-fvsv" 04:11:15.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.123116 6016 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.007s sys 0.004s 04:11:15.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.124374 6016 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:15.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.124920 6016 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:15.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.132097 6016 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s 04:11:15.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.132236 6016 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/wal 04:11:15.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "3295f134662d4295a59d31c8ad94a3e2" 04:11:15.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:15 on dist-test-slave-dist-test-slave-fvsv" 04:11:15.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.132500 6016 fs_report.cc:352] FS layout report 04:11:15.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:15.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/wal 04:11:15.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/wal 04:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/data/data 04:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:15.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:15.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:15.789947 6016 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.633s user 0.251s sys 0.361s 04:11:15.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:15.790235 6016 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.633s user 0.251s sys 0.361s 04:11:15.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.816579 6016 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:15.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.886572 6016 rpc_server.cc:205] RPC server started. Bound to: 127.5.219.62:40075 04:11:15.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.889847 6016 webserver.cc:374] Webserver started at http://127.5.219.62:35111/ using document root and password file 04:11:15.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.890736 6016 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-0/data/info.pb 04:11:15.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.893216 5996 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6016 04:11:15.895 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.895124 6078 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:15.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.5.219.61@KRBTEST.COM; defaulting to no policy 04:11:15.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:15.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.5.219.61@KRBTEST.COM" created. 04:11:15.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.910571 6078 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } has no permanent_uuid. Determining permanent_uuid... 04:11:15.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:15 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.219.62@KRBTEST.COM for kudu/127.5.219.62@KRBTEST.COM 04:11:15.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:15.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.219.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.61.keytab. 04:11:15.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.219.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.61.keytab. 04:11:15.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.219.61 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.61.keytab. 04:11:15.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.219.61 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.61.keytab. 04:11:15.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.938666 5996 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:15.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/wal 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/data 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.219.61:37941 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.219.61 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/logs 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/data/info.pb 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.61.keytab 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.219.61 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:15.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:15.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:15.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.5.219.62:40075,127.5.219.61:37941,127.5.219.60:34321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:15.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.940490 6078 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } has no permanent_uuid. Determining permanent_uuid... 04:11:15.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:15.942181 6078 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.61:37941: Network error: Client connection negotiation failed: client connection to 127.5.219.61:37941: connect: Connection refused (error 111) 04:11:15.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:15.961580 6078 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } attempt: 1 04:11:15.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:15.963390 6078 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.61:37941: Network error: Client connection negotiation failed: client connection to 127.5.219.61:37941: connect: Connection refused (error 111) 04:11:16.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.011734 6078 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } attempt: 2 04:11:16.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:16.013407 6078 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.61:37941: Network error: Client connection negotiation failed: client connection to 127.5.219.61:37941: connect: Connection refused (error 111) 04:11:16.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:16.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:16.090958 6083 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:16.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:16.091341 6083 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.5.219.61 04:11:16.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:16.091593 6083 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:16.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:16.102743 6083 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:16.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:16.102897 6083 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:16.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:16.102991 6083 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:16.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:16.103133 6083 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:16.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:16.103260 6083 flags.cc:404] Enabled experimental flag: --principal=kudu/127.5.219.61 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.121979 6083 master_main.cc:98] Master server non-default flags: 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/data 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/wal 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.5.219.62:40075,127.5.219.61:37941,127.5.219.60:34321 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.219.61:37941 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.61.keytab 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.219.61 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/data/info.pb 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.219.61 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:16.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:16.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:16.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:16.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/logs 04:11:16.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:16.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:16.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:16.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:16.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:16.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:16.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:16.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:16.123 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:16.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.124684 6078 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } attempt: 3 04:11:16.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:16.126267 6078 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.61:37941: Network error: Client connection negotiation failed: client connection to 127.5.219.61:37941: connect: Connection refused (error 111) 04:11:16.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.136718 6083 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 175016us 04:11:16.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:16 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.219.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:16.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.138902 6083 init.cc:359] Logged in from keytab as kudu/127.5.219.61@KRBTEST.COM (short username kudu) 04:11:16.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.141873 6083 fs_manager.cc:263] Metadata directory not provided 04:11:16.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.141996 6083 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:16.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.142228 6083 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:16.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.142329 6083 server_base.cc:449] Attempting to create new FS layout instead 04:11:16.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.145258 6083 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/data/instance: 04:11:16.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "26c68049bd21476b9c250f709c8f1046" 04:11:16.145 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:16 on dist-test-slave-dist-test-slave-fvsv" 04:11:16.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.145951 6083 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/wal/instance: 04:11:16.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "26c68049bd21476b9c250f709c8f1046" 04:11:16.146 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:16 on dist-test-slave-dist-test-slave-fvsv" 04:11:16.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.155576 6083 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.007s sys 0.004s 04:11:16.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.156807 6083 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:16.157 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.157330 6083 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:16.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.164400 6083 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:11:16.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.164523 6083 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/wal 04:11:16.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "26c68049bd21476b9c250f709c8f1046" 04:11:16.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:16 on dist-test-slave-dist-test-slave-fvsv" 04:11:16.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.164746 6083 fs_report.cc:352] FS layout report 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/wal 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/wal 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/data/data 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:16.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:16.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.302561 6078 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } attempt: 4 04:11:16.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:16.304385 6078 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.61:37941: Network error: Client connection negotiation failed: client connection to 127.5.219.61:37941: connect: Connection refused (error 111) 04:11:16.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:16.607692 6078 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } attempt: 5 04:11:16.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:16.609311 6078 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.61:37941: Network error: Client connection negotiation failed: client connection to 127.5.219.61:37941: connect: Connection refused (error 111) 04:11:17.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.040948 6083 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.851s user 0.253s sys 0.574s 04:11:17.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.041193 6083 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.852s user 0.253s sys 0.574s 04:11:17.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.065126 6083 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:17.138 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.138398 6083 rpc_server.cc:205] RPC server started. Bound to: 127.5.219.61:37941 04:11:17.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.141415 6083 webserver.cc:374] Webserver started at http://127.5.219.61:45961/ using document root and password file 04:11:17.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.142292 6083 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-1/data/info.pb 04:11:17.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.143599 6078 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } attempt: 6 04:11:17.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.145795 5996 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6083 04:11:17.152 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.152722 6145 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:17.159 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.5.219.60@KRBTEST.COM; defaulting to no policy 04:11:17.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:17.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.5.219.60@KRBTEST.COM" created. 04:11:17.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:17 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.219.62@KRBTEST.COM for kudu/127.5.219.61@KRBTEST.COM 04:11:17.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.175124 6145 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } has no permanent_uuid. Determining permanent_uuid... 04:11:17.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.186601 6078 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } has no permanent_uuid. Determining permanent_uuid... 04:11:17.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.188357 6078 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.60:34321: Network error: Client connection negotiation failed: client connection to 127.5.219.60:34321: connect: Connection refused (error 111) 04:11:17.190 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:17 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.219.61@KRBTEST.COM for kudu/127.5.219.62@KRBTEST.COM 04:11:17.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:17.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.219.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.60.keytab. 04:11:17.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.219.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.60.keytab. 04:11:17.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.219.60 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.60.keytab. 04:11:17.191 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.5.219.60 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.60.keytab. 04:11:17.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.192581 5996 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:17.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:17.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/wal 04:11:17.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/data 04:11:17.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.219.60:34321 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.219.60 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/logs 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/data/info.pb 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.60.keytab 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.219.60 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:17.193 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.5.219.62:40075,127.5.219.61:37941,127.5.219.60:34321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:17.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.201820 6145 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } has no permanent_uuid. Determining permanent_uuid... 04:11:17.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:17 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.219.61@KRBTEST.COM for kudu/127.5.219.61@KRBTEST.COM 04:11:17.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.218987 6145 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } has no permanent_uuid. Determining permanent_uuid... 04:11:17.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.220619 6145 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.60:34321: Network error: Client connection negotiation failed: client connection to 127.5.219.60:34321: connect: Connection refused (error 111) 04:11:17.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.220710 6078 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } attempt: 1 04:11:17.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.222165 6078 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.60:34321: Network error: Client connection negotiation failed: client connection to 127.5.219.60:34321: connect: Connection refused (error 111) 04:11:17.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.239987 6145 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } attempt: 1 04:11:17.241 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.241580 6145 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.60:34321: Network error: Client connection negotiation failed: client connection to 127.5.219.60:34321: connect: Connection refused (error 111) 04:11:17.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.289841 6145 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } attempt: 2 04:11:17.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.290472 6078 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } attempt: 2 04:11:17.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.291604 6145 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.60:34321: Network error: Client connection negotiation failed: client connection to 127.5.219.60:34321: connect: Connection refused (error 111) 04:11:17.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.292210 6078 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.60:34321: Network error: Client connection negotiation failed: client connection to 127.5.219.60:34321: connect: Connection refused (error 111) 04:11:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.345170 6152 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.345494 6152 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.5.219.60 04:11:17.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.345717 6152 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:17.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.356614 6152 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:17.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.356775 6152 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:17.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.356870 6152 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:17.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.356981 6152 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:17.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.357103 6152 flags.cc:404] Enabled experimental flag: --principal=kudu/127.5.219.60 04:11:17.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.368548 6078 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } attempt: 3 04:11:17.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.370251 6078 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.60:34321: Network error: Client connection negotiation failed: client connection to 127.5.219.60:34321: connect: Connection refused (error 111) 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.376119 6152 master_main.cc:98] Master server non-default flags: 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/data 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/wal 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.5.219.62:40075,127.5.219.61:37941,127.5.219.60:34321 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.5.219.60:34321 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kudu_127.5.219.60.keytab 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.5.219.60 04:11:17.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/data/info.pb 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.5.219.60 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/logs 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:17.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:17.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:17.390 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.390815 6152 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 175516us 04:11:17.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:17 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608677, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.219.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:17.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.393155 6152 init.cc:359] Logged in from keytab as kudu/127.5.219.60@KRBTEST.COM (short username kudu) 04:11:17.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.396601 6152 fs_manager.cc:263] Metadata directory not provided 04:11:17.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.396723 6152 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:17.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.396965 6152 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:17.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.397069 6152 server_base.cc:449] Attempting to create new FS layout instead 04:11:17.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.400028 6152 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/data/instance: 04:11:17.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" 04:11:17.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:17 on dist-test-slave-dist-test-slave-fvsv" 04:11:17.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.400738 6152 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/wal/instance: 04:11:17.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" 04:11:17.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:17 on dist-test-slave-dist-test-slave-fvsv" 04:11:17.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.402946 6145 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } attempt: 3 04:11:17.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.404497 6145 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.60:34321: Network error: Client connection negotiation failed: client connection to 127.5.219.60:34321: connect: Connection refused (error 111) 04:11:17.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.410465 6152 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s 04:11:17.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.411691 6152 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:17.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.412183 6152 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:17.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.418711 6152 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.418839 6152 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/wal 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:17 on dist-test-slave-dist-test-slave-fvsv" 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.419072 6152 fs_report.cc:352] FS layout report 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/wal 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/wal 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/data/data 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:17.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:17.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.507613 6078 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } attempt: 4 04:11:17.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.510023 6078 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.60:34321: Network error: Client connection negotiation failed: client connection to 127.5.219.60:34321: connect: Connection refused (error 111) 04:11:17.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.580798 6145 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } attempt: 4 04:11:17.583 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.583135 6145 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.60:34321: Network error: Client connection negotiation failed: client connection to 127.5.219.60:34321: connect: Connection refused (error 111) 04:11:17.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.793397 6078 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } attempt: 5 04:11:17.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.795850 6078 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.60:34321: Network error: Client connection negotiation failed: client connection to 127.5.219.60:34321: connect: Connection refused (error 111) 04:11:17.886 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:17.886523 6145 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } attempt: 5 04:11:17.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:17.888902 6145 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.5.219.60:34321: Network error: Client connection negotiation failed: client connection to 127.5.219.60:34321: connect: Connection refused (error 111) 04:11:18.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:18.053933 6152 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.607s user 0.001s sys 0.000s 04:11:18.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:18.054133 6152 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.607s user 0.001s sys 0.000s 04:11:18.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.096019 6152 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:18.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.168507 6152 rpc_server.cc:205] RPC server started. Bound to: 127.5.219.60:34321 04:11:18.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.170933 6152 webserver.cc:374] Webserver started at http://127.5.219.60:44777/ using document root and password file 04:11:18.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.171815 6152 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/master-2/data/info.pb 04:11:18.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.175148 6214 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:18.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.176957 5996 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6152 04:11:18.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.188179 6214 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } has no permanent_uuid. Determining permanent_uuid... 04:11:18.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608677, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.219.60@KRBTEST.COM for kudu/127.5.219.62@KRBTEST.COM 04:11:18.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.219.60@KRBTEST.COM 04:11:18.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.216645 6214 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } has no permanent_uuid. Determining permanent_uuid... 04:11:18.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608677, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.219.60@KRBTEST.COM for kudu/127.5.219.61@KRBTEST.COM 04:11:18.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.219.61@KRBTEST.COM 04:11:18.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.238894 6214 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } has no permanent_uuid. Determining permanent_uuid... 04:11:18.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608677, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.219.60@KRBTEST.COM for kudu/127.5.219.60@KRBTEST.COM 04:11:18.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.219.62@KRBTEST.COM 04:11:18.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.259030 5996 external_mini_cluster.cc:628] 0 TS(s) registered with all masters 04:11:18.259 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:18.259 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_kdc_env_vars { } 04:11:18.260 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: get_kdc_env_vars { env_vars { key: "KUDU_ENABLE_KRB5_REALM_FIX" value: "yes" } env_vars { key: "KRB5_KDC_PROFILE" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/kdc.conf" } env_vars { key: "KRB5_CONFIG" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/krb5.conf" } env_vars { key: "KRB5CCNAME" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/krb5cc" } } 04:11:18.260 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_masters { } 04:11:18.261 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.5.219.62" port: 40075 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.5.219.61" port: 37941 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.5.219.60" port: 34321 } } } 04:11:18.261 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_tservers { } 04:11:18.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.261478 6214 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b: Bootstrap starting. 04:11:18.261 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:18.261 [INFO - Time-limited test] (MiniKuduCluster.java:439) Running kinit for user test-admin 04:11:18.261 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: kinit { username: "test-admin" } 04:11:18.265 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.264950 6214 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b: No blocks or log segments found. Creating new log. 04:11:18.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.266366 6214 log.cc:525] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b: Log is configured to *not* fsync() on all Append() calls 04:11:18.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.271956 6214 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b: No bootstrap required, opened a new log 04:11:18.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:18.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:11:18.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.284062 6214 raft_consensus.cc:340] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } 04:11:18.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.284598 6214 raft_consensus.cc:366] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:18.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.284793 6214 raft_consensus.cc:705] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc47c3cc69fe4c06aef13bedf4467b8b, State: Initialized, Role: FOLLOWER 04:11:18.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.285352 6214 consensus_queue.cc:229] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } 04:11:18.285 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:18.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.287298 6214 sys_catalog.cc:339] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } } 04:11:18.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.287881 6214 sys_catalog.cc:342] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [sys.catalog]: This master's current role is: FOLLOWER 04:11:18.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.289100 6214 sys_catalog.cc:427] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [sys.catalog]: configured and running, proceeding with master startup. 04:11:18.290 [DEBUG - Time-limited test] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/krb5cc 04:11:18.292 [DEBUG - Time-limited test] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:11:18.296 [DEBUG - Time-limited test] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster1642518182041757219/krb5kdc/krb5cc 04:11:18.297 [DEBUG - Time-limited test] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:11:18.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.308229 6078 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } attempt: 6 04:11:18.313 [DEBUG - New I/O worker #101] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.5.219.60 04:11:18.313 [DEBUG - New I/O worker #102] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.5.219.61 04:11:18.315 [DEBUG - New I/O worker #100] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.5.219.62 04:11:18.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.219.60@KRBTEST.COM 04:11:18.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.219.61@KRBTEST.COM 04:11:18.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.219.62@KRBTEST.COM 04:11:18.357 [DEBUG - New I/O worker #101] (Negotiator.java:764) Authenticated connection [id: 0x95fb76e6, /127.0.0.1:55008 => /127.5.219.60:34321] using SASL/GSSAPI 04:11:18.354 [DEBUG - New I/O worker #100] (Negotiator.java:764) Authenticated connection [id: 0xf239fe9c, /127.0.0.1:56312 => /127.5.219.62:40075] using SASL/GSSAPI 04:11:18.367 [INFO - New I/O worker #100] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.62:40075: Catalog manager is not initialized. State: Starting 04:11:18.370 [DEBUG - New I/O worker #102] (Negotiator.java:764) Authenticated connection [id: 0xb0db818d, /127.0.0.1:50786 => /127.5.219.61:37941] using SASL/GSSAPI 04:11:18.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608675, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.219.62@KRBTEST.COM for kudu/127.5.219.60@KRBTEST.COM 04:11:18.372 [INFO - New I/O worker #102] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.61:37941: Catalog manager is not initialized. State: Starting 04:11:18.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.287530 6224 sys_catalog.cc:339] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } } 04:11:18.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.400408 6224 sys_catalog.cc:342] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [sys.catalog]: This master's current role is: FOLLOWER 04:11:18.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.410645 6224 raft_consensus.cc:472] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:11:18.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.411113 6224 raft_consensus.cc:494] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } 04:11:18.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.414757 6224 leader_election.cc:296] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3295f134662d4295a59d31c8ad94a3e2 (127.5.219.62:40075), 26c68049bd21476b9c250f709c8f1046 (127.5.219.61:37941) 04:11:18.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.423252 6145 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } attempt: 6 04:11:18.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:18.352089 6170 tablet.cc:1799] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:18.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:18.437186 6231 catalog_manager.cc:1122] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:11:18.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:18.437609 6231 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:11:18.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:18 dist-test-slave-dist-test-slave-fvsv krb5kdc[6005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608676, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.219.61@KRBTEST.COM for kudu/127.5.219.60@KRBTEST.COM 04:11:18.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.444628 6078 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2: Bootstrap starting. 04:11:18.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.448192 6078 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2: No blocks or log segments found. Creating new log. 04:11:18.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.450254 6078 log.cc:525] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2: Log is configured to *not* fsync() on all Append() calls 04:11:18.453 [WARN - New I/O worker #101] (ConnectToCluster.java:281) Unable to find the leader master 127.5.219.62:40075,127.5.219.60:34321,127.5.219.61:37941; will retry 04:11:18.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.457834 6078 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2: No bootstrap required, opened a new log 04:11:18.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.462249 6145 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046: Bootstrap starting. 04:11:18.464 [INFO - New I/O worker #100] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.62:40075: Catalog manager is not initialized. State: Starting 04:11:18.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.471894 6145 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046: No blocks or log segments found. Creating new log. 04:11:18.472 [INFO - New I/O worker #102] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.61:37941: Catalog manager is not initialized. State: Starting 04:11:18.472 [WARN - New I/O worker #102] (ConnectToCluster.java:281) Unable to find the leader master 127.5.219.62:40075,127.5.219.60:34321,127.5.219.61:37941; will retry 04:11:18.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.475641 6078 raft_consensus.cc:340] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } 04:11:18.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.476402 6078 raft_consensus.cc:366] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:18.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.476300 6145 log.cc:525] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046: Log is configured to *not* fsync() on all Append() calls 04:11:18.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.476620 6078 raft_consensus.cc:705] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3295f134662d4295a59d31c8ad94a3e2, State: Initialized, Role: FOLLOWER 04:11:18.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.477295 6078 consensus_queue.cc:229] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } 04:11:18.481 [INFO - New I/O worker #100] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.62:40075: Catalog manager is not initialized. State: Starting 04:11:18.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.481369 6078 sys_catalog.cc:339] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } } 04:11:18.482 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.482177 6078 sys_catalog.cc:342] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [sys.catalog]: This master's current role is: FOLLOWER 04:11:18.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.483454 6078 sys_catalog.cc:427] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [sys.catalog]: configured and running, proceeding with master startup. 04:11:18.486 [INFO - New I/O worker #102] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.61:37941: Catalog manager is not initialized. State: Starting 04:11:18.487 [WARN - New I/O worker #102] (ConnectToCluster.java:281) Unable to find the leader master 127.5.219.62:40075,127.5.219.60:34321,127.5.219.61:37941; will retry 04:11:18.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.487825 6145 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046: No bootstrap required, opened a new log 04:11:18.500 [INFO - New I/O worker #100] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.62:40075: Catalog manager is not initialized. State: Starting 04:11:18.504 [INFO - New I/O worker #102] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.61:37941: Catalog manager is not initialized. State: Starting 04:11:18.505 [WARN - New I/O worker #101] (ConnectToCluster.java:281) Unable to find the leader master 127.5.219.62:40075,127.5.219.60:34321,127.5.219.61:37941; will retry 04:11:18.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.505789 6145 raft_consensus.cc:340] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } 04:11:18.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.506516 6145 raft_consensus.cc:366] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:18.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.506737 6145 raft_consensus.cc:705] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c68049bd21476b9c250f709c8f1046, State: Initialized, Role: FOLLOWER 04:11:18.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.507443 6145 consensus_queue.cc:229] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } 04:11:18.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.510385 6145 sys_catalog.cc:339] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } } 04:11:18.512 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.510977 6145 sys_catalog.cc:342] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [sys.catalog]: This master's current role is: FOLLOWER 04:11:18.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.513099 6145 sys_catalog.cc:427] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [sys.catalog]: configured and running, proceeding with master startup. 04:11:18.542 [INFO - New I/O worker #102] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.61:37941: Catalog manager is not initialized. State: Starting 04:11:18.545 [INFO - New I/O worker #100] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.62:40075: Catalog manager is not initialized. State: Starting 04:11:18.546 [WARN - New I/O worker #100] (ConnectToCluster.java:281) Unable to find the leader master 127.5.219.62:40075,127.5.219.60:34321,127.5.219.61:37941; will retry 04:11:18.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.566258 6055 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3295f134662d4295a59d31c8ad94a3e2" is_pre_election: true 04:11:18.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.481750 6251 sys_catalog.cc:339] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } } 04:11:18.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.567462 6055 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc47c3cc69fe4c06aef13bedf4467b8b in term 0. 04:11:18.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.567514 6251 sys_catalog.cc:342] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [sys.catalog]: This master's current role is: FOLLOWER 04:11:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.569380 6161 leader_election.cc:310] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3295f134662d4295a59d31c8ad94a3e2, dc47c3cc69fe4c06aef13bedf4467b8b; no voters: 04:11:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.572368 6224 raft_consensus.cc:2644] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [term 0 FOLLOWER]: Leader pre-election won for term 1 04:11:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.572620 6224 raft_consensus.cc:472] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:11:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.572827 6224 raft_consensus.cc:2890] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [term 0 FOLLOWER]: Advancing to term 1 04:11:18.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:18.574438 6034 tablet.cc:1799] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:18.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.579732 6224 raft_consensus.cc:494] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } 04:11:18.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.581817 6224 leader_election.cc:296] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [CANDIDATE]: Term 1 election: Requested vote from peers 3295f134662d4295a59d31c8ad94a3e2 (127.5.219.62:40075), 26c68049bd21476b9c250f709c8f1046 (127.5.219.61:37941) 04:11:18.584 [INFO - New I/O worker #102] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.61:37941: Catalog manager is not initialized. State: Starting 04:11:18.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.595384 6055 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3295f134662d4295a59d31c8ad94a3e2" 04:11:18.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.595903 6055 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [term 0 FOLLOWER]: Advancing to term 1 04:11:18.600 [WARN - New I/O worker #100] (ConnectToCluster.java:281) Unable to find the leader master 127.5.219.62:40075,127.5.219.60:34321,127.5.219.61:37941; will retry 04:11:18.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:18.600675 6252 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:11:18.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:18.601125 6252 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:11:18.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.602813 6055 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc47c3cc69fe4c06aef13bedf4467b8b in term 1. 04:11:18.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.604009 6161 leader_election.cc:310] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3295f134662d4295a59d31c8ad94a3e2, dc47c3cc69fe4c06aef13bedf4467b8b; no voters: 04:11:18.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.604825 6224 raft_consensus.cc:2644] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [term 1 FOLLOWER]: Leader election won for term 1 04:11:18.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.606664 6224 raft_consensus.cc:667] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [term 1 LEADER]: Becoming Leader. State: Replica: dc47c3cc69fe4c06aef13bedf4467b8b, State: Running, Role: LEADER 04:11:18.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.607201 6224 consensus_queue.cc:206] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } 04:11:18.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.611431 6256 sys_catalog.cc:339] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc47c3cc69fe4c06aef13bedf4467b8b. Latest consensus state: current_term: 1 leader_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } } 04:11:18.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.612174 6256 sys_catalog.cc:342] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [sys.catalog]: This master's current role is: LEADER 04:11:18.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.616499 6258 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:11:18.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.619940 6258 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:11:18.666 [INFO - New I/O worker #102] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.61:37941: Catalog manager is not initialized. State: Starting 04:11:18.667 [WARN - New I/O worker #102] (ConnectToCluster.java:281) Unable to find the leader master 127.5.219.62:40075,127.5.219.60:34321,127.5.219.61:37941; will retry 04:11:18.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.671412 6055 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [term 1 FOLLOWER]: Refusing update from remote peer dc47c3cc69fe4c06aef13bedf4467b8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:11:18.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.672590 6256 consensus_queue.cc:984] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:11:18.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.688264 6251 sys_catalog.cc:339] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc47c3cc69fe4c06aef13bedf4467b8b. Latest consensus state: current_term: 1 leader_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } } 04:11:18.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.688872 6251 sys_catalog.cc:342] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [sys.catalog]: This master's current role is: FOLLOWER 04:11:18.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.511863 6253 sys_catalog.cc:339] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } } 04:11:18.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.692045 6122 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c68049bd21476b9c250f709c8f1046" is_pre_election: true 04:11:18.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.692646 6253 sys_catalog.cc:342] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [sys.catalog]: This master's current role is: FOLLOWER 04:11:18.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.699208 6259 mvcc.cc:203] Tried to move safe_time back from 6396349147814391808 to 6396349147580305408. Current Snapshot: MvccSnapshot[committed={T|T < 6396349147814391808}] 04:11:18.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.700523 6224 sys_catalog.cc:339] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } } 04:11:18.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.701308 6224 sys_catalog.cc:342] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [sys.catalog]: This master's current role is: LEADER 04:11:18.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.705986 6251 sys_catalog.cc:339] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } } 04:11:18.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.706647 6251 sys_catalog.cc:342] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [sys.catalog]: This master's current role is: FOLLOWER 04:11:18.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.711704 6258 catalog_manager.cc:959] Generated new certificate authority record 04:11:18.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.714152 6258 catalog_manager.cc:1084] Loading token signing keys... 04:11:18.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:18.570972 6101 tablet.cc:1799] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:18.741 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.740921 6258 catalog_manager.cc:4350] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b: Generated new TSK 0 04:11:18.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.583649 6121 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26c68049bd21476b9c250f709c8f1046" 04:11:18.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.669678 6120 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [term 0 FOLLOWER]: Advancing to term 1 04:11:18.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.779945 6120 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [term 1 FOLLOWER]: Refusing update from remote peer dc47c3cc69fe4c06aef13bedf4467b8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:11:18.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.780659 6122 raft_consensus.cc:2325] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc47c3cc69fe4c06aef13bedf4467b8b for term 1 because replica is already servicing an update from a current leader or another vote. 04:11:18.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.782151 6253 raft_consensus.cc:472] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:11:18.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.782411 6253 raft_consensus.cc:494] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } 04:11:18.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.783213 6121 raft_consensus.cc:2325] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dc47c3cc69fe4c06aef13bedf4467b8b for term 1 because replica is already servicing an update from a current leader or another vote. 04:11:18.784 [DEBUG - Hashed wheel timer #12] (AsyncKuduClient.java:2322) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 04:11:18.789 [DEBUG - New I/O worker #100] (Connection.java:691) [peer master-127.5.219.62:40075(127.5.219.62:40075)] cleaning up while in state READY due to: connection disconnected 04:11:18.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.784907 6253 leader_election.cc:296] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3295f134662d4295a59d31c8ad94a3e2 (127.5.219.62:40075), dc47c3cc69fe4c06aef13bedf4467b8b (127.5.219.60:34321) 04:11:18.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.786135 6055 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c68049bd21476b9c250f709c8f1046" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3295f134662d4295a59d31c8ad94a3e2" is_pre_election: true 04:11:18.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.786772 6055 raft_consensus.cc:2311] T 00000000000000000000000000000000 P 3295f134662d4295a59d31c8ad94a3e2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26c68049bd21476b9c250f709c8f1046 for term 2 because replica is either leader or believes a valid leader to be alive. 04:11:18.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.788255 6191 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26c68049bd21476b9c250f709c8f1046" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" is_pre_election: true 04:11:18.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.789798 6191 raft_consensus.cc:2311] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 26c68049bd21476b9c250f709c8f1046 for term 2 because replica is either leader or believes a valid leader to be alive. 04:11:18.791 [DEBUG - New I/O worker #102] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:11:18.791 [DEBUG - New I/O worker #101] (Connection.java:691) [peer master-127.5.219.60:34321(127.5.219.60:34321)] cleaning up while in state READY due to: connection disconnected 04:11:18.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.793051 6092 leader_election.cc:310] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26c68049bd21476b9c250f709c8f1046; no voters: 3295f134662d4295a59d31c8ad94a3e2, dc47c3cc69fe4c06aef13bedf4467b8b 04:11:18.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:18.794225 6093 connection.cc:554] Connection torn down before Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:50786 (request call id 7) could send its response 04:11:18.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.795177 6253 raft_consensus.cc:2592] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority 04:11:18.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.796247 6224 consensus_queue.cc:984] T 00000000000000000000000000000000 P dc47c3cc69fe4c06aef13bedf4467b8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:11:18.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:18.796638 6254 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:11:18.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:18.796854 6254 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:11:18.799 [DEBUG - New I/O worker #102] (Connection.java:691) [peer master-127.5.219.61:37941(127.5.219.61:37941)] cleaning up while in state READY due to: connection disconnected 04:11:18.799 [INFO - New I/O worker #102] (ConnectToCluster.java:361) Unable to connect to master 127.5.219.61:37941: connection disconnected 04:11:18.800 [DEBUG - main] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:11:18.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.823338 5996 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6016 04:11:18.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.841853 6253 sys_catalog.cc:339] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc47c3cc69fe4c06aef13bedf4467b8b. Latest consensus state: current_term: 1 leader_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } } 04:11:18.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.842602 6253 sys_catalog.cc:342] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [sys.catalog]: This master's current role is: FOLLOWER 04:11:18.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.843780 6269 sys_catalog.cc:339] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3295f134662d4295a59d31c8ad94a3e2" member_type: VOTER last_known_addr { host: "127.5.219.62" port: 40075 } } peers { permanent_uuid: "26c68049bd21476b9c250f709c8f1046" member_type: VOTER last_known_addr { host: "127.5.219.61" port: 37941 } } peers { permanent_uuid: "dc47c3cc69fe4c06aef13bedf4467b8b" member_type: VOTER last_known_addr { host: "127.5.219.60" port: 34321 } } } 04:11:18.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.844524 6269 sys_catalog.cc:342] T 00000000000000000000000000000000 P 26c68049bd21476b9c250f709c8f1046 [sys.catalog]: This master's current role is: FOLLOWER 04:11:18.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.875651 5996 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6083 04:11:18.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:18.891213 5996 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6152 04:11:19.928 [INFO - main] (RetryRule.java:136) Creating RetryStatement with result reporter and retry count of 0 (no retries) .04:11:19.929 [INFO - Time-limited test] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskusvpEt/test-tmp 04:11:19.929 [INFO - Time-limited test] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin 04:11:19.929 [INFO - Time-limited test] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskusvpEt/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb] 04:11:19.931 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 0 enable_kerberos: true cluster_root: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247" extra_master_flags: "--authn_token_validity_seconds=10" hms_mode: NONE mini_kdc_options { ticket_lifetime: "10s" renew_lifetime: "20s" } } 04:11:20.106 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:20.106 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_cluster { } 04:11:20.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Loading random data 04:11:20.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Initializing database '/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/principal' for realm 'KRBTEST.COM', 04:11:20.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) master key name 'K/M@KRBTEST.COM' 04:11:20.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:20 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): setting up network... 04:11:20.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:20 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) 04:11:20.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked 04:11:20.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 04:11:20.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:20 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): skipping unrecognized local address family 17 04:11:20.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:20 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): skipping unrecognized local address family 17 04:11:20.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:20 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): set up 1 sockets 04:11:20.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:20 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): commencing operation 04:11:20.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: starting... 04:11:20.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:20.688143 6274 mini_kdc.cc:119] Time spent starting KDC: real 0.555s user 0.001s sys 0.017s 04:11:20.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy 04:11:20.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:20.714 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-admin@KRBTEST.COM" created. 04:11:20.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy 04:11:20.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:20.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-user@KRBTEST.COM" created. 04:11:20.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy 04:11:20.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:20.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "joe-interloper@KRBTEST.COM" created. 04:11:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/test-user.keytab. 04:11:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/test-user.keytab. 04:11:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/test-user.keytab. 04:11:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/test-user.keytab. 04:11:20.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:20 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:20.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:11:20.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.6.32.190@KRBTEST.COM; defaulting to no policy 04:11:20.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:20.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.6.32.190@KRBTEST.COM" created. 04:11:20.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:20.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.6.32.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.190.keytab. 04:11:20.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.6.32.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.190.keytab. 04:11:20.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.6.32.190 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.190.keytab. 04:11:20.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.6.32.190 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.190.keytab. 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:20.844419 6274 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.190:43097 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.190 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:20.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/logs 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/info.pb 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.190.keytab 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.190 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:20.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:20.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:20.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:20.992014 6294 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:20.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:20.992403 6294 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.6.32.190 04:11:20.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:20.992730 6294 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:21.003 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:21.003770 6294 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:21.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:21.003932 6294 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:21.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:21.004083 6294 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:21.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:21.004215 6294 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:21.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:21.004391 6294 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:21.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:21.004566 6294 flags.cc:404] Enabled experimental flag: --principal=kudu/127.6.32.190 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.024343 6294 master_main.cc:98] Master server non-default flags: 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.190:43097 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.190.keytab 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.190 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:21.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/info.pb 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.190 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/logs 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:21.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:21.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.038419 6294 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 177516us 04:11:21.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:21 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608681, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:21.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.040746 6294 init.cc:359] Logged in from keytab as kudu/127.6.32.190@KRBTEST.COM (short username kudu) 04:11:21.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.044479 6294 fs_manager.cc:263] Metadata directory not provided 04:11:21.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.044610 6294 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:21.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.044848 6294 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:21.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.044952 6294 server_base.cc:449] Attempting to create new FS layout instead 04:11:21.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.048005 6294 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/instance: 04:11:21.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "f298b835d8664427a9eca376e52d20f9" 04:11:21.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:21 on dist-test-slave-dist-test-slave-fvsv" 04:11:21.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.048722 6294 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal/instance: 04:11:21.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "f298b835d8664427a9eca376e52d20f9" 04:11:21.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:21 on dist-test-slave-dist-test-slave-fvsv" 04:11:21.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.057651 6294 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.009s sys 0.001s 04:11:21.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.058838 6294 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:21.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.059409 6294 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:21.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.066516 6294 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s 04:11:21.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.066653 6294 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:21.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "f298b835d8664427a9eca376e52d20f9" 04:11:21.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:21 on dist-test-slave-dist-test-slave-fvsv" 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.066872 6294 fs_report.cc:352] FS layout report 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/data 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:21.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:21.812080 6294 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.717s user 0.002s sys 0.000s 04:11:21.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:21.812312 6294 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.717s user 0.002s sys 0.000s 04:11:21.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.855515 6294 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:21.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.929304 6294 rpc_server.cc:205] RPC server started. Bound to: 127.6.32.190:43097 04:11:21.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.931993 6294 webserver.cc:374] Webserver started at http://127.6.32.190:36831/ using document root and password file 04:11:21.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.932878 6294 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/info.pb 04:11:21.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.936919 6356 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:21.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.939288 6274 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6294 04:11:21.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.949447 6356 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } has no permanent_uuid. Determining permanent_uuid... 04:11:21.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.6.32.189@KRBTEST.COM; defaulting to no policy 04:11:21.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:21.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.6.32.189@KRBTEST.COM" created. 04:11:21.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:21 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608681, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.190@KRBTEST.COM for kudu/127.6.32.190@KRBTEST.COM 04:11:21.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:21.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.6.32.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.189.keytab. 04:11:21.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.6.32.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.189.keytab. 04:11:21.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.6.32.189 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.189.keytab. 04:11:21.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.6.32.189 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.189.keytab. 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.982177 6274 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.189:43901 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.189 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:21.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/logs 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/info.pb 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.189.keytab 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.189 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:21.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:21.983165 6356 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } has no permanent_uuid. Determining permanent_uuid... 04:11:21.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:21.984848 6356 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.189:43901: Network error: Client connection negotiation failed: client connection to 127.6.32.189:43901: connect: Connection refused (error 111) 04:11:22.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.004223 6356 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } attempt: 1 04:11:22.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.005843 6356 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.189:43901: Network error: Client connection negotiation failed: client connection to 127.6.32.189:43901: connect: Connection refused (error 111) 04:11:22.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.054160 6356 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } attempt: 2 04:11:22.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.055872 6356 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.189:43901: Network error: Client connection negotiation failed: client connection to 127.6.32.189:43901: connect: Connection refused (error 111) 04:11:22.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:22.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.131603 6361 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:22.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.131963 6361 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.6.32.189 04:11:22.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.132287 6361 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:22.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.143537 6361 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:22.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.143688 6361 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:22.143 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.143824 6361 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:22.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.143954 6361 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:22.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.144134 6361 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:22.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.144302 6361 flags.cc:404] Enabled experimental flag: --principal=kudu/127.6.32.189 04:11:22.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.163400 6361 master_main.cc:98] Master server non-default flags: 04:11:22.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data 04:11:22.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:22.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:22.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:22.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 04:11:22.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:22.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.189:43901 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.189.keytab 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.189 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/info.pb 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.189 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/logs 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:22.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:22.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.167140 6356 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } attempt: 3 04:11:22.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.168720 6356 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.189:43901: Network error: Client connection negotiation failed: client connection to 127.6.32.189:43901: connect: Connection refused (error 111) 04:11:22.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.178128 6361 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 178016us 04:11:22.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:22 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608682, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:22.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.180402 6361 init.cc:359] Logged in from keytab as kudu/127.6.32.189@KRBTEST.COM (short username kudu) 04:11:22.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.183692 6361 fs_manager.cc:263] Metadata directory not provided 04:11:22.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.183962 6361 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:22.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.184345 6361 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:22.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.184463 6361 server_base.cc:449] Attempting to create new FS layout instead 04:11:22.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.187388 6361 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/instance: 04:11:22.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "acd494c7bf49499fab7dec85b5d9b980" 04:11:22.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:22 on dist-test-slave-dist-test-slave-fvsv" 04:11:22.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.188081 6361 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal/instance: 04:11:22.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "acd494c7bf49499fab7dec85b5d9b980" 04:11:22.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:22 on dist-test-slave-dist-test-slave-fvsv" 04:11:22.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.196733 6361 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s 04:11:22.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.198079 6361 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:22.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.198551 6361 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:22.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.205597 6361 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s 04:11:22.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.205718 6361 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:22.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "acd494c7bf49499fab7dec85b5d9b980" 04:11:22.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:22 on dist-test-slave-dist-test-slave-fvsv" 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.205919 6361 fs_report.cc:352] FS layout report 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/data 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:22.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:22.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.345064 6356 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } attempt: 4 04:11:22.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.347483 6356 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.189:43901: Network error: Client connection negotiation failed: client connection to 127.6.32.189:43901: connect: Connection refused (error 111) 04:11:22.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.650893 6356 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } attempt: 5 04:11:22.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.653339 6356 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.189:43901: Network error: Client connection negotiation failed: client connection to 127.6.32.189:43901: connect: Connection refused (error 111) 04:11:22.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.902469 6361 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.672s user 0.284s sys 0.364s 04:11:22.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:22.902732 6361 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.672s user 0.284s sys 0.364s 04:11:22.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:22.945762 6361 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:23.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.019199 6361 rpc_server.cc:205] RPC server started. Bound to: 127.6.32.189:43901 04:11:23.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.022500 6361 webserver.cc:374] Webserver started at http://127.6.32.189:42789/ using document root and password file 04:11:23.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.023680 6361 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/info.pb 04:11:23.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.027382 6423 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:23.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.029934 6274 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6361 04:11:23.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.6.32.188@KRBTEST.COM; defaulting to no policy 04:11:23.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.041121 6423 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } has no permanent_uuid. Determining permanent_uuid... 04:11:23.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:23.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.6.32.188@KRBTEST.COM" created. 04:11:23.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:23 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608682, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.189@KRBTEST.COM for kudu/127.6.32.190@KRBTEST.COM 04:11:23.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.069083 6423 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } has no permanent_uuid. Determining permanent_uuid... 04:11:23.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:23.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.6.32.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.188.keytab. 04:11:23.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.6.32.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.188.keytab. 04:11:23.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.6.32.188 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.188.keytab. 04:11:23.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.6.32.188 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.188.keytab. 04:11:23.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.070715 6274 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.188:36321 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.188 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/logs 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/info.pb 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:23.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:23.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:23.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:23.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.188.keytab 04:11:23.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.188 04:11:23.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:23.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:23.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:23.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:23.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:23.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:23.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:23 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608682, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.189@KRBTEST.COM for kudu/127.6.32.189@KRBTEST.COM 04:11:23.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.092587 6423 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } has no permanent_uuid. Determining permanent_uuid... 04:11:23.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.094074 6423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.188:36321: Network error: Client connection negotiation failed: client connection to 127.6.32.188:36321: connect: Connection refused (error 111) 04:11:23.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.113445 6423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } attempt: 1 04:11:23.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.115011 6423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.188:36321: Network error: Client connection negotiation failed: client connection to 127.6.32.188:36321: connect: Connection refused (error 111) 04:11:23.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.163363 6423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } attempt: 2 04:11:23.165 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.165102 6423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.188:36321: Network error: Client connection negotiation failed: client connection to 127.6.32.188:36321: connect: Connection refused (error 111) 04:11:23.188 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.187724 6356 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } attempt: 6 04:11:23.201 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:23 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608681, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.190@KRBTEST.COM for kudu/127.6.32.189@KRBTEST.COM 04:11:23.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.210290 6356 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } has no permanent_uuid. Determining permanent_uuid... 04:11:23.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.211865 6356 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.188:36321: Network error: Client connection negotiation failed: client connection to 127.6.32.188:36321: connect: Connection refused (error 111) 04:11:23.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:23.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.224423 6429 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:23.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.224725 6429 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.6.32.188 04:11:23.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.224951 6429 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:23.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.236153 6429 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:23.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.236297 6429 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:23.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.236403 6429 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:23.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.236498 6429 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:23.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.236614 6429 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:23.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.236737 6429 flags.cc:404] Enabled experimental flag: --principal=kudu/127.6.32.188 04:11:23.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.244199 6356 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } attempt: 1 04:11:23.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.245827 6356 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.188:36321: Network error: Client connection negotiation failed: client connection to 127.6.32.188:36321: connect: Connection refused (error 111) 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.255220 6429 master_main.cc:98] Master server non-default flags: 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.188:36321 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.188.keytab 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.188 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:23.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/info.pb 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.188 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/logs 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:23.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:23.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.269196 6429 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 178516us 04:11:23.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:23 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608683, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:23.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.271412 6429 init.cc:359] Logged in from keytab as kudu/127.6.32.188@KRBTEST.COM (short username kudu) 04:11:23.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.274163 6429 fs_manager.cc:263] Metadata directory not provided 04:11:23.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.274302 6429 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:23.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.274600 6429 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:23.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.274734 6429 server_base.cc:449] Attempting to create new FS layout instead 04:11:23.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.276432 6423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } attempt: 3 04:11:23.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.278007 6423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.188:36321: Network error: Client connection negotiation failed: client connection to 127.6.32.188:36321: connect: Connection refused (error 111) 04:11:23.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.278021 6429 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/instance: 04:11:23.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "6713f258192f4403a1e3c5d663ca5280" 04:11:23.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:23 on dist-test-slave-dist-test-slave-fvsv" 04:11:23.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.278856 6429 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal/instance: 04:11:23.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "6713f258192f4403a1e3c5d663ca5280" 04:11:23.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:23 on dist-test-slave-dist-test-slave-fvsv" 04:11:23.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.287801 6429 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s 04:11:23.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.289093 6429 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:23.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.289620 6429 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:23.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.296304 6429 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s 04:11:23.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.296433 6429 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:23.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "6713f258192f4403a1e3c5d663ca5280" 04:11:23.296 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:23 on dist-test-slave-dist-test-slave-fvsv" 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.296708 6429 fs_report.cc:352] FS layout report 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/data 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:23.297 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:23.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.314149 6356 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } attempt: 2 04:11:23.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.315762 6356 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.188:36321: Network error: Client connection negotiation failed: client connection to 127.6.32.188:36321: connect: Connection refused (error 111) 04:11:23.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.392037 6356 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } attempt: 3 04:11:23.393 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.393764 6356 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.188:36321: Network error: Client connection negotiation failed: client connection to 127.6.32.188:36321: connect: Connection refused (error 111) 04:11:23.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.454282 6423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } attempt: 4 04:11:23.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.455868 6423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.188:36321: Network error: Client connection negotiation failed: client connection to 127.6.32.188:36321: connect: Connection refused (error 111) 04:11:23.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.531101 6356 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } attempt: 4 04:11:23.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.532783 6356 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.188:36321: Network error: Client connection negotiation failed: client connection to 127.6.32.188:36321: connect: Connection refused (error 111) 04:11:23.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.759141 6423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } attempt: 5 04:11:23.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.760831 6423 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.188:36321: Network error: Client connection negotiation failed: client connection to 127.6.32.188:36321: connect: Connection refused (error 111) 04:11:23.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:23.816059 6356 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } attempt: 5 04:11:23.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:23.817700 6356 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.6.32.188:36321: Network error: Client connection negotiation failed: client connection to 127.6.32.188:36321: connect: Connection refused (error 111) 04:11:24.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:24.155700 6429 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.833s user 0.329s sys 0.503s 04:11:24.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:24.155896 6429 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.834s user 0.329s sys 0.503s 04:11:24.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.220305 6429 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:24.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.288810 6429 rpc_server.cc:205] RPC server started. Bound to: 127.6.32.188:36321 04:11:24.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.291182 6429 webserver.cc:374] Webserver started at http://127.6.32.188:38409/ using document root and password file 04:11:24.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.292062 6429 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/info.pb 04:11:24.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.295346 6492 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:24.295 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.295146 6423 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } attempt: 6 04:11:24.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.298014 6274 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6429 04:11:24.318 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.316900 6492 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } has no permanent_uuid. Determining permanent_uuid... 04:11:24.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:24 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608682, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.189@KRBTEST.COM for kudu/127.6.32.188@KRBTEST.COM 04:11:24.327 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:24 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.32.188@KRBTEST.COM 04:11:24.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.329989 6356 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } attempt: 6 04:11:24.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:24 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608683, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.188@KRBTEST.COM for kudu/127.6.32.190@KRBTEST.COM 04:11:24.342 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:24 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608681, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.190@KRBTEST.COM for kudu/127.6.32.188@KRBTEST.COM 04:11:24.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.354279 6423 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Bootstrap starting. 04:11:24.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.358186 6423 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: No blocks or log segments found. Creating new log. 04:11:24.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.360476 6423 log.cc:525] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Log is configured to *not* fsync() on all Append() calls 04:11:24.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.360394 6492 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } has no permanent_uuid. Determining permanent_uuid... 04:11:24.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.369400 6356 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: Bootstrap starting. 04:11:24.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.373862 6356 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: No blocks or log segments found. Creating new log. 04:11:24.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:24 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.32.189@KRBTEST.COM 04:11:24.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.376065 6356 log.cc:525] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: Log is configured to *not* fsync() on all Append() calls 04:11:24.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.379021 6423 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: No bootstrap required, opened a new log 04:11:24.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.380014 6356 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: No bootstrap required, opened a new log 04:11:24.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:24 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608683, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.188@KRBTEST.COM for kudu/127.6.32.189@KRBTEST.COM 04:11:24.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.398324 6423 raft_consensus.cc:340] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:24.399 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.399073 6423 raft_consensus.cc:366] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:24.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.399284 6423 raft_consensus.cc:705] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd494c7bf49499fab7dec85b5d9b980, State: Initialized, Role: FOLLOWER 04:11:24.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.398854 6356 raft_consensus.cc:340] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:24.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.399569 6356 raft_consensus.cc:366] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:24.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.399756 6356 raft_consensus.cc:705] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f298b835d8664427a9eca376e52d20f9, State: Initialized, Role: FOLLOWER 04:11:24.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.399700 6492 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } has no permanent_uuid. Determining permanent_uuid... 04:11:24.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.400064 6423 consensus_queue.cc:229] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:24.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.400478 6356 consensus_queue.cc:229] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:24.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.405148 6423 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.406802 6423 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: FOLLOWER 04:11:24.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:24 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608683, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.188@KRBTEST.COM for kudu/127.6.32.188@KRBTEST.COM 04:11:24.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.408179 6423 sys_catalog.cc:427] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: configured and running, proceeding with master startup. 04:11:24.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:24 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.32.190@KRBTEST.COM 04:11:24.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.410945 6506 sys_catalog.cc:339] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.411718 6506 sys_catalog.cc:342] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: This master's current role is: FOLLOWER 04:11:24.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.425525 6274 external_mini_cluster.cc:628] 0 TS(s) registered with all masters 04:11:24.425 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:24.426 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_kdc_env_vars { } 04:11:24.427 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: get_kdc_env_vars { env_vars { key: "KRB5_CONFIG" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf" } env_vars { key: "KRB5CCNAME" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc" } env_vars { key: "KUDU_ENABLE_KRB5_REALM_FIX" value: "yes" } env_vars { key: "KRB5_KDC_PROFILE" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf" } } 04:11:24.427 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_masters { } 04:11:24.428 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.6.32.190" port: 43097 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.6.32.189" port: 43901 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.6.32.188" port: 36321 } } } 04:11:24.428 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: get_tservers { } 04:11:24.429 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:24.429 [INFO - Time-limited test] (MiniKuduCluster.java:439) Running kinit for user test-admin 04:11:24.429 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: kinit { username: "test-admin" } 04:11:24.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.444577 6492 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Bootstrap starting. 04:11:24.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.448972 6492 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: No blocks or log segments found. Creating new log. 04:11:24.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.450928 6492 log.cc:525] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Log is configured to *not* fsync() on all Append() calls 04:11:24.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.465116 6492 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: No bootstrap required, opened a new log 04:11:24.474 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:24 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:24.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:11:24.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.485954 6492 raft_consensus.cc:340] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:24.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.486733 6492 raft_consensus.cc:366] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:24.487 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.487035 6492 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6713f258192f4403a1e3c5d663ca5280, State: Initialized, Role: FOLLOWER 04:11:24.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.487720 6492 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:24.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.490713 6492 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.491351 6492 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:24.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.492456 6492 sys_catalog.cc:427] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: configured and running, proceeding with master startup. 04:11:24.493 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:24.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.405355 6505 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.506 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.506212 6505 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: FOLLOWER 04:11:24.521 [DEBUG - Time-limited test] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc 04:11:24.522 [DEBUG - Time-limited test] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:11:24.523 [DEBUG - Time-limited test] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc 04:11:24.524 [DEBUG - Time-limited test] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:11:24.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.419391 6356 sys_catalog.cc:339] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.537456 6356 sys_catalog.cc:342] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: This master's current role is: FOLLOWER 04:11:24.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.538759 6356 sys_catalog.cc:427] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: configured and running, proceeding with master startup. 04:11:24.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:24.549147 6524 catalog_manager.cc:1122] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:11:24.557 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:24.549404 6524 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:11:24.559 [DEBUG - Time-limited test] (SecurityContext.java:153) Using caller-provided subject with Kerberos principal test-admin@KRBTEST.COM. Caller is responsible for refreshing credentials. 04:11:24.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:24.452127 6379 tablet.cc:1799] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:24.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:24.572127 6507 catalog_manager.cc:1122] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:11:24.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:24.572309 6507 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:11:24.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:24.612484 6312 tablet.cc:1799] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:24.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.494330 6511 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.650009 6511 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:24.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.650482 6511 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:11:24.652 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.650895 6511 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:24.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.654251 6511 leader_election.cc:296] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097), acd494c7bf49499fab7dec85b5d9b980 (127.6.32.189:43901) 04:11:24.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.655388 6333 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6713f258192f4403a1e3c5d663ca5280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f298b835d8664427a9eca376e52d20f9" is_pre_election: true 04:11:24.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.656155 6333 raft_consensus.cc:2350] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6713f258192f4403a1e3c5d663ca5280 in term 0. 04:11:24.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.656183 6400 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6713f258192f4403a1e3c5d663ca5280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd494c7bf49499fab7dec85b5d9b980" is_pre_election: true 04:11:24.657 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.656941 6400 raft_consensus.cc:2350] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6713f258192f4403a1e3c5d663ca5280 in term 0. 04:11:24.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.658093 6441 leader_election.cc:310] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6713f258192f4403a1e3c5d663ca5280, f298b835d8664427a9eca376e52d20f9; no voters: 04:11:24.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.658753 6511 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 0 FOLLOWER]: Leader pre-election won for term 1 04:11:24.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.659232 6511 raft_consensus.cc:472] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:11:24.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.659607 6511 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 0 FOLLOWER]: Advancing to term 1 04:11:24.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:24.661120 6512 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:11:24.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:24.661360 6512 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:11:24.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.664162 6511 raft_consensus.cc:494] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:24.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.666147 6333 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6713f258192f4403a1e3c5d663ca5280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f298b835d8664427a9eca376e52d20f9" 04:11:24.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.666612 6333 raft_consensus.cc:2890] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 0 FOLLOWER]: Advancing to term 1 04:11:24.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.672219 6333 raft_consensus.cc:2350] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6713f258192f4403a1e3c5d663ca5280 in term 1. 04:11:24.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.673292 6441 leader_election.cc:310] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6713f258192f4403a1e3c5d663ca5280, f298b835d8664427a9eca376e52d20f9; no voters: 04:11:24.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.674103 6534 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 1 FOLLOWER]: Leader election won for term 1 04:11:24.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.675297 6511 leader_election.cc:296] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [CANDIDATE]: Term 1 election: Requested vote from peers f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097), acd494c7bf49499fab7dec85b5d9b980 (127.6.32.189:43901) 04:11:24.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.676362 6400 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6713f258192f4403a1e3c5d663ca5280" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acd494c7bf49499fab7dec85b5d9b980" 04:11:24.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.676939 6400 raft_consensus.cc:2890] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 0 FOLLOWER]: Advancing to term 1 04:11:24.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.682165 6400 raft_consensus.cc:2350] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6713f258192f4403a1e3c5d663ca5280 in term 1. 04:11:24.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.687685 6534 raft_consensus.cc:667] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 1 LEADER]: Becoming Leader. State: Replica: 6713f258192f4403a1e3c5d663ca5280, State: Running, Role: LEADER 04:11:24.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.688527 6534 consensus_queue.cc:206] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:24.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.694754 6511 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6713f258192f4403a1e3c5d663ca5280. Latest consensus state: current_term: 1 leader_uuid: "6713f258192f4403a1e3c5d663ca5280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.695271 6511 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: LEADER 04:11:24.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.701440 6538 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:11:24.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.704664 6538 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:11:24.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:24.725867 6448 tablet.cc:1799] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:24.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.765733 6400 raft_consensus.cc:1184] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 1 FOLLOWER]: Refusing update from remote peer 6713f258192f4403a1e3c5d663ca5280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:11:24.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.765736 6333 raft_consensus.cc:1184] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 1 FOLLOWER]: Refusing update from remote peer 6713f258192f4403a1e3c5d663ca5280: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:11:24.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.767550 6534 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:11:24.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.768205 6511 consensus_queue.cc:984] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:11:24.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.786473 6506 sys_catalog.cc:339] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6713f258192f4403a1e3c5d663ca5280. Latest consensus state: current_term: 1 leader_uuid: "6713f258192f4403a1e3c5d663ca5280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.787153 6506 sys_catalog.cc:342] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: This master's current role is: FOLLOWER 04:11:24.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.791543 6539 mvcc.cc:203] Tried to move safe_time back from 6396349172765765632 to 6396349172495048704. Current Snapshot: MvccSnapshot[committed={T|T < 6396349172765765632}] 04:11:24.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.796722 6505 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6713f258192f4403a1e3c5d663ca5280. Latest consensus state: current_term: 1 leader_uuid: "6713f258192f4403a1e3c5d663ca5280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.797430 6505 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: FOLLOWER 04:11:24.797 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.795433 6540 mvcc.cc:203] Tried to move safe_time back from 6396349172765765632 to 6396349172495048704. Current Snapshot: MvccSnapshot[committed={T|T < 6396349172765765632}] 04:11:24.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.798144 6536 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6713f258192f4403a1e3c5d663ca5280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.798867 6536 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: LEADER 04:11:24.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.799614 6538 catalog_manager.cc:959] Generated new certificate authority record 04:11:24.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.798774 6506 sys_catalog.cc:339] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6713f258192f4403a1e3c5d663ca5280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.800458 6506 sys_catalog.cc:342] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: This master's current role is: FOLLOWER 04:11:24.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.801815 6538 catalog_manager.cc:1084] Loading token signing keys... 04:11:24.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.803743 6536 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6713f258192f4403a1e3c5d663ca5280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.804383 6536 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: LEADER 04:11:24.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.808225 6541 mvcc.cc:203] Tried to move safe_time back from 6396349172765765632 to 6396349172495048704. Current Snapshot: MvccSnapshot[committed={T|T < 6396349172765765632}] 04:11:24.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.814340 6505 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6713f258192f4403a1e3c5d663ca5280" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:24.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.815250 6505 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: FOLLOWER 04:11:24.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:24.834581 6538 catalog_manager.cc:4350] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Generated new TSK 0 04:11:25.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:25.556447 6524 catalog_manager.cc:1120] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: acquiring CA information for follower catalog manager: success 04:11:25.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:25.559685 6524 catalog_manager.cc:1148] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:25.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:25.578426 6507 catalog_manager.cc:1120] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: acquiring CA information for follower catalog manager: success 04:11:25.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:25.581240 6507 catalog_manager.cc:1148] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:27.560 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.6.32.189:43901 04:11:27.560 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:11:27.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.561036 6274 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6361 04:11:27.579 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:27.579 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.6.32.188:36321 04:11:27.580 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:11:27.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.580353 6274 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6429 04:11:27.598 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:27.599 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.6.32.190:43097 04:11:27.599 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:11:27.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.599570 6274 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6294 04:11:27.618 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:27.618 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.6.32.189:43901 04:11:27.619 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:11:27.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.620277 6274 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:27.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:27.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:27.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data 04:11:27.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:27.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:27.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:27.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:27.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.189:43901 04:11:27.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.189 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=42789 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/logs 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/info.pb 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.189.keytab 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.189 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:27.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:27.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:27.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:27.767887 6547 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:27.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:27.768230 6547 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.6.32.189 04:11:27.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:27.768460 6547 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:27.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:27.779446 6547 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:27.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:27.779590 6547 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:27.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:27.779700 6547 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:27.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:27.779793 6547 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:27.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:27.779907 6547 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:27.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:27.780027 6547 flags.cc:404] Enabled experimental flag: --principal=kudu/127.6.32.189 04:11:27.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.798522 6547 master_main.cc:98] Master server non-default flags: 04:11:27.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data 04:11:27.798 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.189:43901 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.189.keytab 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.189 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/info.pb 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.189 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=42789 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/logs 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:27.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:27.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:27.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:27.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:27.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.812832 6547 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 180516us 04:11:27.814 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:27 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608687, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:27.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.815143 6547 init.cc:359] Logged in from keytab as kudu/127.6.32.189@KRBTEST.COM (short username kudu) 04:11:27.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.818941 6547 fs_manager.cc:263] Metadata directory not provided 04:11:27.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.819128 6547 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:27.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.824539 6547 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s 04:11:27.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.825203 6547 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:27.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.825747 6547 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:27.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.832890 6547 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:11:27.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.833025 6547 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:27.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "acd494c7bf49499fab7dec85b5d9b980" 04:11:27.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:22 on dist-test-slave-dist-test-slave-fvsv" 04:11:27.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:27.834355 6547 fs_report.cc:352] FS layout report 04:11:27.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:27.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:27.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:27.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/data 04:11:27.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:27.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:27.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:27.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:27.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:27.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:27.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:27.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:27.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:27.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:28.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:28.507457 6547 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.644s user 0.002s sys 0.000s 04:11:28.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:28.507678 6547 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.644s user 0.002s sys 0.000s 04:11:28.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.550714 6547 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:28.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.623514 6547 rpc_server.cc:205] RPC server started. Bound to: 127.6.32.189:43901 04:11:28.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.626267 6547 webserver.cc:374] Webserver started at http://127.6.32.189:42789/ using document root and password file 04:11:28.627 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.627228 6547 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/info.pb 04:11:28.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.634116 6609 sys_catalog.cc:188] Verifying existing consensus state 04:11:28.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.635224 6274 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6547 04:11:28.635 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:28.635 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.6.32.188:36321 04:11:28.635 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:11:28.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.636644 6274 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:28.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:28.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.188:36321 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.188 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=38409 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/logs 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/info.pb 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:28.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:28.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:28.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:28.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.188.keytab 04:11:28.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.188 04:11:28.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:28.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:28.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:28.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:28.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:28.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:28.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.638314 6609 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Bootstrap starting. 04:11:28.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.665884 6609 log.cc:525] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Log is configured to *not* fsync() on all Append() calls 04:11:28.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.674363 6609 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:11:28.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.675068 6609 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Bootstrap complete. 04:11:28.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.694876 6609 raft_consensus.cc:340] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:28.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.700131 6609 raft_consensus.cc:705] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd494c7bf49499fab7dec85b5d9b980, State: Initialized, Role: FOLLOWER 04:11:28.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.700788 6609 consensus_queue.cc:229] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.702965 6612 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:28.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.703567 6612 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: FOLLOWER 04:11:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.703970 6609 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:28.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.704372 6609 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: FOLLOWER 04:11:28.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.705230 6609 sys_catalog.cc:427] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: configured and running, proceeding with master startup. 04:11:28.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.731812 6613 catalog_manager.cc:1120] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: acquiring CA information for follower catalog manager: success 04:11:28.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.734652 6613 catalog_manager.cc:1148] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:28.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:28.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:28.792037 6610 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:28.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:28.792368 6610 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.6.32.188 04:11:28.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:28.792603 6610 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:28.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:28.803948 6610 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:28.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:28.804095 6610 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:28.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:28.804201 6610 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:28.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:28.804292 6610 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:28.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:28.804400 6610 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:28.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:28.804518 6610 flags.cc:404] Enabled experimental flag: --principal=kudu/127.6.32.188 04:11:28.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.824002 6610 master_main.cc:98] Master server non-default flags: 04:11:28.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data 04:11:28.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:28.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:28.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:28.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 04:11:28.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:28.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:28.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.188:36321 04:11:28.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:28.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.188.keytab 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.188 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/info.pb 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.188 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=38409 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/logs 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:28.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:28.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:28.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:28.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:28.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:28.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:28.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:28.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:28.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.838201 6610 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 181016us 04:11:28.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:28 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608688, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:28.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.840734 6610 init.cc:359] Logged in from keytab as kudu/127.6.32.188@KRBTEST.COM (short username kudu) 04:11:28.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.847730 6610 fs_manager.cc:263] Metadata directory not provided 04:11:28.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.847888 6610 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:28.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.856493 6610 fs_manager.cc:399] Time spent opening directory manager: real 0.007s user 0.004s sys 0.004s 04:11:28.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.857223 6610 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:28.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.857764 6610 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:28.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.868404 6610 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s 04:11:28.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.868541 6610 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:28.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "6713f258192f4403a1e3c5d663ca5280" 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:23 on dist-test-slave-dist-test-slave-fvsv" 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:28.869884 6610 fs_report.cc:352] FS layout report 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/data 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:28.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:29.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:29.613145 6610 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.710s user 0.000s sys 0.002s 04:11:29.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:29.613442 6610 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.710s user 0.000s sys 0.002s 04:11:29.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.664297 6610 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:29.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.810030 6610 rpc_server.cc:205] RPC server started. Bound to: 127.6.32.188:36321 04:11:29.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.812695 6610 webserver.cc:374] Webserver started at http://127.6.32.188:38409/ using document root and password file 04:11:29.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.813639 6610 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/info.pb 04:11:29.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.815459 6274 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6610 04:11:29.815 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:29.816 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.6.32.190:43097 04:11:29.816 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.817868 6274 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.190:43097 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.190 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=36831 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:29.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/logs 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/info.pb 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.190.keytab 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.190 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:29.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:29.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.822736 6676 sys_catalog.cc:188] Verifying existing consensus state 04:11:29.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.826830 6676 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Bootstrap starting. 04:11:29.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.880080 6676 log.cc:525] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Log is configured to *not* fsync() on all Append() calls 04:11:29.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.901082 6676 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:11:29.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.901942 6676 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Bootstrap complete. 04:11:29.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.933315 6676 raft_consensus.cc:340] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:29.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.941764 6676 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6713f258192f4403a1e3c5d663ca5280, State: Initialized, Role: FOLLOWER 04:11:29.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.942592 6676 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:29.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.947896 6676 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:29.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.948572 6676 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:29.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.949669 6676 sys_catalog.cc:427] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: configured and running, proceeding with master startup. 04:11:30.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:30.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.006752 6677 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:30.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.007129 6677 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.6.32.190 04:11:30.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.007462 6677 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:30.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.019587 6677 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:30.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.019748 6677 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:30.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.019899 6677 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:30.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.020052 6677 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:30.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.020215 6677 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:30.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.020380 6677 flags.cc:404] Enabled experimental flag: --principal=kudu/127.6.32.190 04:11:30.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.043105 6677 master_main.cc:98] Master server non-default flags: 04:11:30.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data 04:11:30.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:30.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:30.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:30.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 04:11:30.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:30.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:30.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.190:43097 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.190.keytab 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.190 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/info.pb 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.190 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=36831 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/logs 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:30.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:30.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:30.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:30.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:30.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:30.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:30.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:30.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:29.948264 6679 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:30.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.053376 6679 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:30.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.058866 6677 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 182016us 04:11:30.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:30 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608690, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:30.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.061480 6677 init.cc:359] Logged in from keytab as kudu/127.6.32.190@KRBTEST.COM (short username kudu) 04:11:30.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.066164 6680 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: acquiring CA information for follower catalog manager: success 04:11:30.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.068125 6677 fs_manager.cc:263] Metadata directory not provided 04:11:30.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.068323 6677 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:30.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.069023 6680 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:30.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.081015 6677 fs_manager.cc:399] Time spent opening directory manager: real 0.009s user 0.002s sys 0.001s 04:11:30.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.081940 6677 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:30.082 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.082665 6677 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:30.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.095146 6677 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s 04:11:30.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.095331 6677 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:30.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "f298b835d8664427a9eca376e52d20f9" 04:11:30.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:21 on dist-test-slave-dist-test-slave-fvsv" 04:11:30.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.097254 6677 fs_report.cc:352] FS layout report 04:11:30.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:30.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:30.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:30.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/data 04:11:30.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:30.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:30.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:30.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:30.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:30.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:30.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:30.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:30.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:30.098 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:30.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.420109 6695 raft_consensus.cc:472] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:11:30.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.420583 6695 raft_consensus.cc:494] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:30.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.431731 6558 leader_election.cc:341] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097): Network error: Client connection negotiation failed: client connection to 127.6.32.190:43097: connect: Connection refused (error 111) 04:11:30.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.454128 6653 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acd494c7bf49499fab7dec85b5d9b980" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6713f258192f4403a1e3c5d663ca5280" is_pre_election: true 04:11:30.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.455013 6653 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd494c7bf49499fab7dec85b5d9b980 in term 1. 04:11:30.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.456660 6558 leader_election.cc:310] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6713f258192f4403a1e3c5d663ca5280, acd494c7bf49499fab7dec85b5d9b980; no voters: f298b835d8664427a9eca376e52d20f9 04:11:30.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.432257 6695 leader_election.cc:296] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097), 6713f258192f4403a1e3c5d663ca5280 (127.6.32.188:36321) 04:11:30.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.547037 6695 raft_consensus.cc:2644] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 1 FOLLOWER]: Leader pre-election won for term 2 04:11:30.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.548264 6695 raft_consensus.cc:472] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:11:30.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.548552 6695 raft_consensus.cc:2890] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 1 FOLLOWER]: Advancing to term 2 04:11:30.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.556186 6695 raft_consensus.cc:494] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:30.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.557862 6695 leader_election.cc:296] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 2 election: Requested vote from peers f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097), 6713f258192f4403a1e3c5d663ca5280 (127.6.32.188:36321) 04:11:30.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.558938 6558 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.32.188:36321, user_credentials={real_user=slave}} blocked reactor thread for 102291us 04:11:30.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.560868 6558 leader_election.cc:341] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097): Network error: Client connection negotiation failed: client connection to 127.6.32.190:43097: connect: Connection refused (error 111) 04:11:30.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.561990 6653 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acd494c7bf49499fab7dec85b5d9b980" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6713f258192f4403a1e3c5d663ca5280" 04:11:30.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.562505 6653 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 1 FOLLOWER]: Advancing to term 2 04:11:30.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.568665 6653 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd494c7bf49499fab7dec85b5d9b980 in term 2. 04:11:30.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.569664 6558 leader_election.cc:310] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6713f258192f4403a1e3c5d663ca5280, acd494c7bf49499fab7dec85b5d9b980; no voters: f298b835d8664427a9eca376e52d20f9 04:11:30.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.570358 6699 raft_consensus.cc:2644] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 2 FOLLOWER]: Leader election won for term 2 04:11:30.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.570732 6699 raft_consensus.cc:667] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 2 LEADER]: Becoming Leader. State: Replica: acd494c7bf49499fab7dec85b5d9b980, State: Running, Role: LEADER 04:11:30.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.571312 6699 consensus_queue.cc:206] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:30.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.580389 6695 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd494c7bf49499fab7dec85b5d9b980. Latest consensus state: current_term: 2 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:30.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.581092 6695 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: LEADER 04:11:30.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.586313 6701 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:11:30.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.590219 6701 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:11:30.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.592828 6701 catalog_manager.cc:1084] Loading token signing keys... 04:11:30.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.594627 6701 catalog_manager.cc:4361] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Loaded TSK: 0 04:11:30.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.971894 6653 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 2 FOLLOWER]: Refusing update from remote peer acd494c7bf49499fab7dec85b5d9b980: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) 04:11:30.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.973589 6695 consensus_queue.cc:984] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s 04:11:30.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:30.986570 6558 consensus_peers.cc:458] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 -> Peer f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097): Couldn't send request to peer f298b835d8664427a9eca376e52d20f9. Status: Network error: Client connection negotiation failed: client connection to 127.6.32.190:43097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:11:30.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.988301 6702 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd494c7bf49499fab7dec85b5d9b980. Latest consensus state: current_term: 2 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:30.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.989053 6702 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:30.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.996320 6699 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:30.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:30.997035 6699 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: LEADER 04:11:31.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.000008 6702 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:31.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.000675 6702 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:31.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:31.151206 6677 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 1.016s user 0.000s sys 0.002s 04:11:31.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:31.151477 6677 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 1.016s user 0.001s sys 0.002s 04:11:31.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.216311 6677 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:31.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.351814 6677 rpc_server.cc:205] RPC server started. Bound to: 127.6.32.190:43097 04:11:31.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.355829 6677 webserver.cc:374] Webserver started at http://127.6.32.190:36831/ using document root and password file 04:11:31.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.357064 6677 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/info.pb 04:11:31.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.364720 6274 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6677 04:11:31.365 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:31.377 [WARN - New I/O worker #118] (SecurityContext.java:201) Caller-provided Subject has a Kerberos ticket that is about to expire. Kudu expects the application to renew or re-acquire its own tickets before expiration. 04:11:31.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.377436 6755 sys_catalog.cc:188] Verifying existing consensus state 04:11:31.378 [DEBUG - New I/O worker #118] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.6.32.189 04:11:31.384 [DEBUG - New I/O worker #119] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.6.32.188 04:11:31.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.391084 6755 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: Bootstrap starting. 04:11:31.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:31 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.32.188@KRBTEST.COM 04:11:31.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:31 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.32.189@KRBTEST.COM 04:11:31.411 [DEBUG - New I/O worker #119] (Negotiator.java:764) Authenticated connection [id: 0x019bf58e, /127.0.0.1:53246 => /127.6.32.188:36321] using SASL/GSSAPI 04:11:31.416 [DEBUG - New I/O worker #120] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.6.32.190 04:11:31.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:31 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608687, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.189@KRBTEST.COM for kudu/127.6.32.190@KRBTEST.COM 04:11:31.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:31 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.32.190@KRBTEST.COM 04:11:31.435 [DEBUG - New I/O worker #118] (Negotiator.java:764) Authenticated connection [id: 0x3b4d18e8, /127.0.0.1:45142 => /127.6.32.189:43901] using SASL/GSSAPI 04:11:31.439 [DEBUG - New I/O worker #120] (Negotiator.java:764) Authenticated connection [id: 0x13dfa6e8, /127.0.0.1:60510 => /127.6.32.190:43097] using SASL/GSSAPI 04:11:31.444 [DEBUG - New I/O worker #118] (AsyncKuduClient.java:2322) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 04:11:31.445 [DEBUG - New I/O worker #118] (AsyncKuduClient.java:1296) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=30000, elapsed=80), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.6.32.189:43901, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.6.32.188:36321, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.6.32.190:43097, [50ms] Sub RPC ConnectToMaster: received response from server master-127.6.32.188:36321: OK, [77ms] Sub RPC ConnectToMaster: received response from server master-127.6.32.189:43901: OK, [78ms] Sub RPC ConnectToMaster: received response from server master-127.6.32.190:43097: Service unavailable: Catalog manager is not initialized. State: Starting (error 0), deferred=null) after lookup 04:11:31.445 [INFO - New I/O worker #120] (ConnectToCluster.java:361) Unable to connect to master 127.6.32.190:43097: Catalog manager is not initialized. State: Starting 04:11:31.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.448374 6755 log.cc:525] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: Log is configured to *not* fsync() on all Append() calls 04:11:31.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.460897 6755 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:11:31.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.461696 6755 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: Bootstrap complete. 04:11:31.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.481907 6755 raft_consensus.cc:340] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:31.490 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.490502 6755 raft_consensus.cc:705] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f298b835d8664427a9eca376e52d20f9, State: Initialized, Role: FOLLOWER 04:11:31.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.491292 6755 consensus_queue.cc:229] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:31.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.495914 6755 sys_catalog.cc:339] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:31.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.496647 6755 sys_catalog.cc:342] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: This master's current role is: FOLLOWER 04:11:31.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.500031 6755 sys_catalog.cc:427] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: configured and running, proceeding with master startup. 04:11:31.579 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.575814 6732 raft_consensus.cc:2890] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 1 FOLLOWER]: Advancing to term 2 04:11:31.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.496146 6762 sys_catalog.cc:339] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:31.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.580101 6762 sys_catalog.cc:342] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: This master's current role is: FOLLOWER 04:11:31.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.590236 6762 sys_catalog.cc:339] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd494c7bf49499fab7dec85b5d9b980. Latest consensus state: current_term: 2 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:31.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.589534 6766 sys_catalog.cc:339] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:31.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.590984 6762 sys_catalog.cc:342] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: This master's current role is: FOLLOWER 04:11:31.591 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.591014 6766 sys_catalog.cc:342] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: This master's current role is: FOLLOWER 04:11:31.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.599257 6763 catalog_manager.cc:1120] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: acquiring CA information for follower catalog manager: success 04:11:31.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.602033 6763 catalog_manager.cc:1148] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:31.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.873276 6699 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:31.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:31.874027 6699 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: LEADER 04:11:34.501 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.6.32.189:43901 04:11:34.501 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:11:34.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.502255 6274 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6547 04:11:34.521 [DEBUG - New I/O worker #118] (Connection.java:691) [peer master-127.6.32.189:43901(127.6.32.189:43901)] cleaning up while in state READY due to: connection disconnected 04:11:34.522 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:34.523 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.6.32.188:36321 04:11:34.523 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:11:34.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.523484 6274 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6610 04:11:34.542 [DEBUG - New I/O worker #119] (Connection.java:691) [peer master-127.6.32.188:36321(127.6.32.188:36321)] cleaning up while in state READY due to: connection disconnected 04:11:34.542 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:34.543 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.6.32.190:43097 04:11:34.543 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:11:34.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.543529 6274 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6677 04:11:34.561 [DEBUG - New I/O worker #120] (Connection.java:691) [peer master-127.6.32.190:43097(127.6.32.190:43097)] cleaning up while in state READY due to: connection disconnected 04:11:34.561 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:34.562 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.6.32.189:43901 04:11:34.562 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:11:34.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.563405 6274 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:34.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:34.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:34.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data 04:11:34.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:34.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:34.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:34.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:34.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.189:43901 04:11:34.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.189 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=42789 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/logs 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/info.pb 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.189.keytab 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.189 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:34.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:34.715994 6772 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:34.716455 6772 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.6.32.189 04:11:34.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:34.716687 6772 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:34.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:34.727594 6772 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:34.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:34.727735 6772 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:34.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:34.727850 6772 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:34.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:34.727946 6772 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:34.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:34.728058 6772 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:34.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:34.728179 6772 flags.cc:404] Enabled experimental flag: --principal=kudu/127.6.32.189 04:11:34.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.746608 6772 master_main.cc:98] Master server non-default flags: 04:11:34.746 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.189:43901 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.189.keytab 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.189 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/info.pb 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.189 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=42789 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/logs 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:34.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:34.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:34.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:34.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:34.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:34.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.760792 6772 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 184016us 04:11:34.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:34 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:34.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.763098 6772 init.cc:359] Logged in from keytab as kudu/127.6.32.189@KRBTEST.COM (short username kudu) 04:11:34.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.765779 6772 fs_manager.cc:263] Metadata directory not provided 04:11:34.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.765920 6772 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:34.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.771173 6772 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s 04:11:34.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.771878 6772 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:34.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.772464 6772 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:34.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.779109 6772 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:11:34.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.779256 6772 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:34.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "acd494c7bf49499fab7dec85b5d9b980" 04:11:34.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:22 on dist-test-slave-dist-test-slave-fvsv" 04:11:34.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:34.780572 6772 fs_report.cc:352] FS layout report 04:11:34.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/data 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:34.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:35.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:35.424345 6772 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.620s user 0.256s sys 0.363s 04:11:35.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:35.424634 6772 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.620s user 0.256s sys 0.363s 04:11:35.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.483660 6772 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:35.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.553268 6772 rpc_server.cc:205] RPC server started. Bound to: 127.6.32.189:43901 04:11:35.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.555641 6772 webserver.cc:374] Webserver started at http://127.6.32.189:42789/ using document root and password file 04:11:35.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.556555 6772 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/info.pb 04:11:35.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.561005 6274 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6772 04:11:35.561 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:35.561 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.6.32.188:36321 04:11:35.561 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:11:35.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.562763 6274 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.188:36321 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.188 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=38409 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/logs 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/info.pb 04:11:35.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:35.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:35.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:35.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:35.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.188.keytab 04:11:35.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.188 04:11:35.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:35.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:35.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:35.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:35.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:35.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:35.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.564075 6834 sys_catalog.cc:188] Verifying existing consensus state 04:11:35.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.567764 6834 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Bootstrap starting. 04:11:35.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.595285 6834 log.cc:525] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Log is configured to *not* fsync() on all Append() calls 04:11:35.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.603843 6834 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:11:35.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.604429 6834 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Bootstrap complete. 04:11:35.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.624053 6834 raft_consensus.cc:340] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:35.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.629285 6834 raft_consensus.cc:705] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd494c7bf49499fab7dec85b5d9b980, State: Initialized, Role: FOLLOWER 04:11:35.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.629853 6834 consensus_queue.cc:229] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:35.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.631945 6837 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:35.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.632448 6837 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: FOLLOWER 04:11:35.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.632761 6834 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:35.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.633146 6834 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: FOLLOWER 04:11:35.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.633908 6834 sys_catalog.cc:427] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: configured and running, proceeding with master startup. 04:11:35.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.635345 6838 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:11:35.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:35.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:35.711691 6835 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:35.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:35.712059 6835 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.6.32.188 04:11:35.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:35.712383 6835 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:35.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:35.723896 6835 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:35.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:35.724052 6835 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:35.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:35.724203 6835 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:35.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:35.724349 6835 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:35.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:35.724519 6835 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:35.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:35.724684 6835 flags.cc:404] Enabled experimental flag: --principal=kudu/127.6.32.188 04:11:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.743302 6835 master_main.cc:98] Master server non-default flags: 04:11:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data 04:11:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:35.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.188:36321 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.188.keytab 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.188 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/info.pb 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.188 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=38409 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/logs 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:35.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:35.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:35.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:35.758 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.758179 6835 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 184516us 04:11:35.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:35 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608695, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:35.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.760426 6835 init.cc:359] Logged in from keytab as kudu/127.6.32.188@KRBTEST.COM (short username kudu) 04:11:35.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.763561 6835 fs_manager.cc:263] Metadata directory not provided 04:11:35.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.763685 6835 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:35.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.768512 6835 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s 04:11:35.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.769121 6835 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:35.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.769608 6835 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:35.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.776373 6835 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:11:35.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.776505 6835 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:35.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "6713f258192f4403a1e3c5d663ca5280" 04:11:35.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:23 on dist-test-slave-dist-test-slave-fvsv" 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:35.777787 6835 fs_report.cc:352] FS layout report 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/data 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:35.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:36.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:36.411588 6835 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.610s user 0.194s sys 0.415s 04:11:36.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:36.411859 6835 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.610s user 0.194s sys 0.415s 04:11:36.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.478870 6835 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:36.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.556432 6835 rpc_server.cc:205] RPC server started. Bound to: 127.6.32.188:36321 04:11:36.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.559264 6835 webserver.cc:374] Webserver started at http://127.6.32.188:38409/ using document root and password file 04:11:36.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.560196 6835 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/info.pb 04:11:36.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.566555 6274 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6835 04:11:36.566 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:36.567 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.6.32.190:43097 04:11:36.567 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:11:36.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.567883 6900 sys_catalog.cc:188] Verifying existing consensus state 04:11:36.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.568392 6274 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:36.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:36.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:36.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data 04:11:36.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:36.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:36.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:36.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.190:43097 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.190 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=36831 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/logs 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/info.pb 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.190.keytab 04:11:36.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.190 04:11:36.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:36.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:36.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:36.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:36.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:36.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:36.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.572445 6900 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Bootstrap starting. 04:11:36.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.600270 6900 log.cc:525] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Log is configured to *not* fsync() on all Append() calls 04:11:36.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.609079 6900 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:11:36.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.609721 6900 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Bootstrap complete. 04:11:36.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.629942 6900 raft_consensus.cc:340] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:36.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.635164 6900 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6713f258192f4403a1e3c5d663ca5280, State: Initialized, Role: FOLLOWER 04:11:36.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.635874 6900 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:36.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.638075 6903 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:36.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.638582 6903 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:36.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.639001 6900 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:36.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.639467 6900 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:36.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.640301 6900 sys_catalog.cc:427] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: configured and running, proceeding with master startup. 04:11:36.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.641742 6905 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:11:36.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.644676 6838 catalog_manager.cc:1120] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: acquiring CA information for follower catalog manager: success 04:11:36.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.646733 6838 catalog_manager.cc:1148] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:36.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:36.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:36.761669 6901 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:36.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:36.762125 6901 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.6.32.190 04:11:36.762 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:36.762496 6901 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:36.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:36.781864 6901 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:36.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:36.782075 6901 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:36.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:36.782253 6901 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:36.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:36.782413 6901 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:36.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:36.782608 6901 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:36.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:36.782809 6901 flags.cc:404] Enabled experimental flag: --principal=kudu/127.6.32.190 04:11:36.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.815531 6901 master_main.cc:98] Master server non-default flags: 04:11:36.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.190:43097 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.190.keytab 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.190 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/info.pb 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.190 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=36831 04:11:36.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/logs 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:36.817 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:36.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.830463 6901 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 185016us 04:11:36.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:36 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608696, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:36.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.833087 6901 init.cc:359] Logged in from keytab as kudu/127.6.32.190@KRBTEST.COM (short username kudu) 04:11:36.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.837148 6901 fs_manager.cc:263] Metadata directory not provided 04:11:36.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.837291 6901 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:36.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.844461 6901 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s 04:11:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.845157 6901 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:36.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.845696 6901 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:36.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.856415 6901 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s 04:11:36.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.856555 6901 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:36.857 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "f298b835d8664427a9eca376e52d20f9" 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:21 on dist-test-slave-dist-test-slave-fvsv" 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:36.857869 6901 fs_report.cc:352] FS layout report 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/data 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:36.858 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:37.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.048872 6918 raft_consensus.cc:472] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:11:37.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.176043 6918 raft_consensus.cc:494] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:37.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.178903 6918 leader_election.cc:296] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097), 6713f258192f4403a1e3c5d663ca5280 (127.6.32.188:36321) 04:11:37.189 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:37.189100 6783 leader_election.cc:341] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097): Network error: Client connection negotiation failed: client connection to 127.6.32.190:43097: connect: Connection refused (error 111) 04:11:37.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:37 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.189@KRBTEST.COM for kudu/127.6.32.188@KRBTEST.COM 04:11:37.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.234184 6877 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acd494c7bf49499fab7dec85b5d9b980" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6713f258192f4403a1e3c5d663ca5280" is_pre_election: true 04:11:37.236 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.235481 6877 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd494c7bf49499fab7dec85b5d9b980 in term 2. 04:11:37.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.237923 6783 leader_election.cc:310] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6713f258192f4403a1e3c5d663ca5280, acd494c7bf49499fab7dec85b5d9b980; no voters: f298b835d8664427a9eca376e52d20f9 04:11:37.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.239722 6918 raft_consensus.cc:2644] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 2 FOLLOWER]: Leader pre-election won for term 3 04:11:37.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.240476 6918 raft_consensus.cc:472] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:11:37.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.240730 6918 raft_consensus.cc:2890] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 2 FOLLOWER]: Advancing to term 3 04:11:37.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.250275 6918 raft_consensus.cc:494] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:37.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:37.253281 6783 leader_election.cc:341] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097): Network error: Client connection negotiation failed: client connection to 127.6.32.190:43097: connect: Connection refused (error 111) 04:11:37.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.255555 6877 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acd494c7bf49499fab7dec85b5d9b980" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6713f258192f4403a1e3c5d663ca5280" 04:11:37.259 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.256114 6877 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 2 FOLLOWER]: Advancing to term 3 04:11:37.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.263130 6918 leader_election.cc:296] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 3 election: Requested vote from peers f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097), 6713f258192f4403a1e3c5d663ca5280 (127.6.32.188:36321) 04:11:37.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.264107 6877 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd494c7bf49499fab7dec85b5d9b980 in term 3. 04:11:37.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.265264 6783 leader_election.cc:310] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6713f258192f4403a1e3c5d663ca5280, acd494c7bf49499fab7dec85b5d9b980; no voters: f298b835d8664427a9eca376e52d20f9 04:11:37.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.266434 6918 raft_consensus.cc:2644] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 3 FOLLOWER]: Leader election won for term 3 04:11:37.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.275198 6918 raft_consensus.cc:667] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 3 LEADER]: Becoming Leader. State: Replica: acd494c7bf49499fab7dec85b5d9b980, State: Running, Role: LEADER 04:11:37.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.276026 6918 consensus_queue.cc:206] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:37.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.284212 6922 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd494c7bf49499fab7dec85b5d9b980. Latest consensus state: current_term: 3 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:37.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.285199 6922 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: LEADER 04:11:37.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.287766 6924 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:11:37.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.290590 6924 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:11:37.294 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.294596 6924 catalog_manager.cc:1084] Loading token signing keys... 04:11:37.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.300837 6924 catalog_manager.cc:4361] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Loaded TSK: 0 04:11:37.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:37.603036 6901 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.718s user 0.289s sys 0.374s 04:11:37.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:37.603600 6901 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.719s user 0.290s sys 0.374s 04:11:37.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.654748 6905 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: acquiring CA information for follower catalog manager: success 04:11:37.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.656764 6905 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:37.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.680064 6901 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:37.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:37.775974 6783 consensus_peers.cc:458] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 -> Peer f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097): Couldn't send request to peer f298b835d8664427a9eca376e52d20f9. Status: Network error: Client connection negotiation failed: client connection to 127.6.32.190:43097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:11:37.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.780040 6877 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 3 FOLLOWER]: Refusing update from remote peer acd494c7bf49499fab7dec85b5d9b980: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) 04:11:37.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.929354 6922 consensus_queue.cc:984] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s 04:11:37.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.943915 6967 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd494c7bf49499fab7dec85b5d9b980. Latest consensus state: current_term: 3 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:37.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.944941 6967 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:37.950 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.950037 6967 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:37.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.950852 6967 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:37.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.950589 6971 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:37.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.951474 6971 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: LEADER 04:11:37.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.971787 6901 rpc_server.cc:205] RPC server started. Bound to: 127.6.32.190:43097 04:11:37.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.974263 6901 webserver.cc:374] Webserver started at http://127.6.32.190:36831/ using document root and password file 04:11:37.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.975327 6901 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/info.pb 04:11:37.977 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:37.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.976974 6274 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6901 04:11:37.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.983476 6983 sys_catalog.cc:188] Verifying existing consensus state 04:11:37.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:37.989064 6983 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: Bootstrap starting. 04:11:37.999 [DEBUG - New I/O worker #121] (Negotiator.java:764) Authenticated connection [id: 0x9a944823, /127.0.0.1:45156 => /127.6.32.189:43901] using TOKEN/null 04:11:38.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.022579 6983 log.cc:525] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: Log is configured to *not* fsync() on all Append() calls 04:11:38.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.031960 6983 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:11:38.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.032646 6983 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: Bootstrap complete. 04:11:38.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.052959 6983 raft_consensus.cc:340] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:38.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.058281 6983 raft_consensus.cc:705] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f298b835d8664427a9eca376e52d20f9, State: Initialized, Role: FOLLOWER 04:11:38.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.059128 6983 consensus_queue.cc:229] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:38.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.061414 6986 sys_catalog.cc:339] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.061945 6986 sys_catalog.cc:342] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: This master's current role is: FOLLOWER 04:11:38.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.062350 6983 sys_catalog.cc:339] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:38.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.062772 6983 sys_catalog.cc:342] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: This master's current role is: FOLLOWER 04:11:38.063 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.063761 6983 sys_catalog.cc:427] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: configured and running, proceeding with master startup. 04:11:38.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.065189 6987 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting 04:11:38.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:38 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608694, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.189@KRBTEST.COM for kudu/127.6.32.190@KRBTEST.COM 04:11:38.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.298322 6953 raft_consensus.cc:2890] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [term 2 FOLLOWER]: Advancing to term 3 04:11:38.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.316104 6991 sys_catalog.cc:339] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:38.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.320974 6991 sys_catalog.cc:342] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: This master's current role is: FOLLOWER 04:11:38.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.316589 6986 sys_catalog.cc:339] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd494c7bf49499fab7dec85b5d9b980. Latest consensus state: current_term: 3 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:38.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.419415 6986 sys_catalog.cc:342] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9 [sys.catalog]: This master's current role is: FOLLOWER 04:11:38.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.694309 6922 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:38.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:38.695173 6922 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: LEADER 04:11:39.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:39.073441 6987 catalog_manager.cc:1120] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: acquiring CA information for follower catalog manager: success 04:11:39.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:39.076366 6987 catalog_manager.cc:1148] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:41.057 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.6.32.189:43901 04:11:41.057 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 1 } } 04:11:41.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.058470 6274 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6772 04:11:41.078 [DEBUG - New I/O worker #121] (Connection.java:691) [peer master-127.6.32.189:43901(127.6.32.189:43901)] cleaning up while in state READY due to: connection disconnected 04:11:41.079 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:41.079 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.6.32.188:36321 04:11:41.079 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 2 } } 04:11:41.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.079900 6274 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6835 04:11:41.098 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:41.098 [INFO - Time-limited test] (MiniKuduCluster.java:330) Killing master server 127.6.32.190:43097 04:11:41.098 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: stop_daemon { id { type: MASTER index: 0 } } 04:11:41.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.099031 6274 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6901 04:11:41.116 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:41.116 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.6.32.189:43901 04:11:41.116 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 1 } } 04:11:41.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.117686 6274 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.189:43901 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.189 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=42789 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/logs 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/info.pb 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:41.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:41.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:41.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.189.keytab 04:11:41.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.189 04:11:41.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:41.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:41.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:41.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:41.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:41.119 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:41.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:41.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:41.267674 6997 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:41.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:41.268013 6997 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.6.32.189 04:11:41.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:41.268241 6997 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:41.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:41.279264 6997 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:41.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:41.279409 6997 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:41.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:41.279515 6997 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:41.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:41.279605 6997 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:41.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:41.279740 6997 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:41.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:41.279868 6997 flags.cc:404] Enabled experimental flag: --principal=kudu/127.6.32.189 04:11:41.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.298586 6997 master_main.cc:98] Master server non-default flags: 04:11:41.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data 04:11:41.298 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.189:43901 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.189.keytab 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.189 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/info.pb 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.189 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=42789 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/logs 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:41.299 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:41.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:41.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:41.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:41.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.312724 6997 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 187516us 04:11:41.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:41 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:41.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.314944 6997 init.cc:359] Logged in from keytab as kudu/127.6.32.189@KRBTEST.COM (short username kudu) 04:11:41.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.317745 6997 fs_manager.cc:263] Metadata directory not provided 04:11:41.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.317873 6997 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:41.322 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.322702 6997 fs_manager.cc:399] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s 04:11:41.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.323361 6997 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:41.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.323876 6997 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:41.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.330485 6997 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s 04:11:41.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.330610 6997 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:41.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "acd494c7bf49499fab7dec85b5d9b980" 04:11:41.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:22 on dist-test-slave-dist-test-slave-fvsv" 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.331904 6997 fs_report.cc:352] FS layout report 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/wal 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/data 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:41.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:41.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:41.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:41.962468 6997 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.606s user 0.206s sys 0.399s 04:11:41.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:41.962685 6997 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.607s user 0.206s sys 0.399s 04:11:41.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:41.996269 6997 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:42.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.069900 6997 rpc_server.cc:205] RPC server started. Bound to: 127.6.32.189:43901 04:11:42.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.073221 6997 webserver.cc:374] Webserver started at http://127.6.32.189:42789/ using document root and password file 04:11:42.074 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.074205 6997 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-1/data/info.pb 04:11:42.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.080693 6274 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 6997 04:11:42.081 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:42.081 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.6.32.188:36321 04:11:42.081 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 2 } } 04:11:42.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.082818 6274 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:42.083 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:42.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:42.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data 04:11:42.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:42.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:42.084 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.188:36321 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.188 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=38409 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/logs 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/info.pb 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:42.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:42.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:42.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.188.keytab 04:11:42.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.188 04:11:42.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:42.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:42.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:42.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:42.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:42.086 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:42.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.085767 7059 sys_catalog.cc:188] Verifying existing consensus state 04:11:42.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.100261 7059 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Bootstrap starting. 04:11:42.154 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.154407 7059 log.cc:525] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Log is configured to *not* fsync() on all Append() calls 04:11:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.173559 7059 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:11:42.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.174383 7059 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Bootstrap complete. 04:11:42.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.204916 7059 raft_consensus.cc:340] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:42.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.213166 7059 raft_consensus.cc:705] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: acd494c7bf49499fab7dec85b5d9b980, State: Initialized, Role: FOLLOWER 04:11:42.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.213985 7059 consensus_queue.cc:229] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:42.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.219944 7059 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:42.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.220588 7059 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: FOLLOWER 04:11:42.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.221813 7059 sys_catalog.cc:427] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: configured and running, proceeding with master startup. 04:11:42.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.220278 7062 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:42.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.292137 7062 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: FOLLOWER 04:11:42.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.308410 7063 catalog_manager.cc:1120] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: acquiring CA information for follower catalog manager: success 04:11:42.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.315454 7063 catalog_manager.cc:1148] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:42.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:42.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:42.346057 7060 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:42.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:42.346536 7060 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.6.32.188 04:11:42.347 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:42.346905 7060 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:42.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:42.366237 7060 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:42.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:42.366439 7060 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:42.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:42.366613 7060 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:42.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:42.366776 7060 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:42.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:42.366961 7060 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:42.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:42.367194 7060 flags.cc:404] Enabled experimental flag: --principal=kudu/127.6.32.188 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.399969 7060 master_main.cc:98] Master server non-default flags: 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.188:36321 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.188.keytab 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.188 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:42.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/info.pb 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.188 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=38409 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/logs 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:42.401 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:42.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.423749 7060 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 188016us 04:11:42.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:42 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608702, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:42.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.426648 7060 init.cc:359] Logged in from keytab as kudu/127.6.32.188@KRBTEST.COM (short username kudu) 04:11:42.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.435890 7060 fs_manager.cc:263] Metadata directory not provided 04:11:42.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.436030 7060 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:42.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.444537 7060 fs_manager.cc:399] Time spent opening directory manager: real 0.006s user 0.007s sys 0.001s 04:11:42.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.445271 7060 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:42.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.445808 7060 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:42.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.456286 7060 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.000s sys 0.005s 04:11:42.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.456418 7060 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:42.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "6713f258192f4403a1e3c5d663ca5280" 04:11:42.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:23 on dist-test-slave-dist-test-slave-fvsv" 04:11:42.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:42.457736 7060 fs_report.cc:352] FS layout report 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/wal 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/data 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:42.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:43.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.211622 7060 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.720s user 0.002s sys 0.000s 04:11:43.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.211890 7060 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.721s user 0.003s sys 0.000s 04:11:43.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.256475 7060 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:43.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.431802 7060 rpc_server.cc:205] RPC server started. Bound to: 127.6.32.188:36321 04:11:43.436 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.435854 7060 webserver.cc:374] Webserver started at http://127.6.32.188:38409/ using document root and password file 04:11:43.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.437104 7060 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-2/data/info.pb 04:11:43.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.439371 6274 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 7060 04:11:43.439 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:43.439 [INFO - Time-limited test] (MiniKuduCluster.java:311) Starting master server 127.6.32.190:43097 04:11:43.439 [DEBUG - Time-limited test] (MiniKuduCluster.java:155) Request: start_daemon { id { type: MASTER index: 0 } } 04:11:43.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.440790 6274 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.190:43097 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.190 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=36831 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/logs 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/info.pb 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:43.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:43.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:43.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:43.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.190.keytab 04:11:43.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.190 04:11:43.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:43.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:43.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:43.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:43.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:43.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:43.448 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.448081 7126 sys_catalog.cc:188] Verifying existing consensus state 04:11:43.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.453433 7126 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Bootstrap starting. 04:11:43.492 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.492123 7126 log.cc:525] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Log is configured to *not* fsync() on all Append() calls 04:11:43.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.504288 7126 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates 04:11:43.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.505102 7126 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: Bootstrap complete. 04:11:43.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.535071 7126 raft_consensus.cc:340] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:43.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.542820 7126 raft_consensus.cc:705] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6713f258192f4403a1e3c5d663ca5280, State: Initialized, Role: FOLLOWER 04:11:43.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.543601 7126 consensus_queue.cc:229] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:43.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.546747 7126 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:43.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.547415 7126 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:43.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.552501 7126 sys_catalog.cc:427] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: configured and running, proceeding with master startup. 04:11:43.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.546995 7129 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:43.656 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.655918 7129 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:43.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.673884 7130 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: acquiring CA information for follower catalog manager: success 04:11:43.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.680754 7130 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:43.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:43.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.750257 7127 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:43.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.750721 7127 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.6.32.190 04:11:43.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.751124 7127 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:43.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.770526 7127 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:43.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.770738 7127 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:43.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.770915 7127 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:43.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.771107 7127 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:43.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.771322 7127 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:43.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.771519 7127 flags.cc:404] Enabled experimental flag: --principal=kudu/127.6.32.190 04:11:43.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.807622 7127 master_main.cc:98] Master server non-default flags: 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.6.32.190:43097,127.6.32.189:43901,127.6.32.188:36321 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.6.32.190:43097 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/krb5kdc/kudu_127.6.32.190.keytab 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.6.32.190 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:43.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/info.pb 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.6.32.190 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=36831 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/logs 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:43.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:43.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.831672 7127 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 188516us 04:11:43.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:43 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608703, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:43.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.835002 7127 init.cc:359] Logged in from keytab as kudu/127.6.32.190@KRBTEST.COM (short username kudu) 04:11:43.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.839464 7127 fs_manager.cc:263] Metadata directory not provided 04:11:43.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.839812 7127 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:43.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.847180 7127 fs_manager.cc:399] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s 04:11:43.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.848270 7127 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:43.849 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.849112 7127 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:43.872 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.872718 7127 fs_manager.cc:419] Time spent opening block manager: real 0.015s user 0.004s sys 0.010s 04:11:43.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.873206 7127 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:43.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "f298b835d8664427a9eca376e52d20f9" 04:11:43.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:21 on dist-test-slave-dist-test-slave-fvsv" 04:11:43.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.875475 7127 fs_report.cc:352] FS layout report 04:11:43.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/wal 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/data 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:43.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:43.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.955590 7143 raft_consensus.cc:472] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:11:43.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.956094 7143 raft_consensus.cc:494] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:43.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.967761 7143 leader_election.cc:296] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097), 6713f258192f4403a1e3c5d663ca5280 (127.6.32.188:36321) 04:11:43.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.971457 7008 leader_election.cc:341] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097): Network error: Client connection negotiation failed: client connection to 127.6.32.190:43097: connect: Connection refused (error 111) 04:11:43.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.985319 7103 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acd494c7bf49499fab7dec85b5d9b980" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6713f258192f4403a1e3c5d663ca5280" is_pre_election: true 04:11:43.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.986102 7103 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acd494c7bf49499fab7dec85b5d9b980 in term 3. 04:11:43.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.987897 7008 leader_election.cc:310] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6713f258192f4403a1e3c5d663ca5280, acd494c7bf49499fab7dec85b5d9b980; no voters: f298b835d8664427a9eca376e52d20f9 04:11:43.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.989042 7143 raft_consensus.cc:2644] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 3 FOLLOWER]: Leader pre-election won for term 4 04:11:43.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.989269 7143 raft_consensus.cc:472] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:11:43.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.989465 7143 raft_consensus.cc:2890] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 3 FOLLOWER]: Advancing to term 4 04:11:43.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.994733 7143 raft_consensus.cc:494] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:43.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:43.997076 7008 leader_election.cc:341] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097): Network error: Client connection negotiation failed: client connection to 127.6.32.190:43097: connect: Connection refused (error 111) 04:11:43.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.998603 7103 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "acd494c7bf49499fab7dec85b5d9b980" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6713f258192f4403a1e3c5d663ca5280" 04:11:43.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:43.999107 7103 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 3 FOLLOWER]: Advancing to term 4 04:11:44.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.002122 7143 leader_election.cc:296] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 4 election: Requested vote from peers f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097), 6713f258192f4403a1e3c5d663ca5280 (127.6.32.188:36321) 04:11:44.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.004571 7103 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acd494c7bf49499fab7dec85b5d9b980 in term 4. 04:11:44.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.006587 7008 leader_election.cc:310] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6713f258192f4403a1e3c5d663ca5280, acd494c7bf49499fab7dec85b5d9b980; no voters: f298b835d8664427a9eca376e52d20f9 04:11:44.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.007777 7143 raft_consensus.cc:2644] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 4 FOLLOWER]: Leader election won for term 4 04:11:44.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.009820 7143 raft_consensus.cc:667] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [term 4 LEADER]: Becoming Leader. State: Replica: acd494c7bf49499fab7dec85b5d9b980, State: Running, Role: LEADER 04:11:44.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.010618 7143 consensus_queue.cc:206] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } 04:11:44.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.018041 7148 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd494c7bf49499fab7dec85b5d9b980. Latest consensus state: current_term: 4 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:44.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.018929 7148 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: LEADER 04:11:44.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.021435 7150 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:11:44.025 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.025672 7150 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:11:44.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.028654 7150 catalog_manager.cc:1084] Loading token signing keys... 04:11:44.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.030861 7150 catalog_manager.cc:4361] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980: Loaded TSK: 0 04:11:44.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:44.528580 7008 consensus_peers.cc:458] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 -> Peer f298b835d8664427a9eca376e52d20f9 (127.6.32.190:43097): Couldn't send request to peer f298b835d8664427a9eca376e52d20f9. Status: Network error: Client connection negotiation failed: client connection to 127.6.32.190:43097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. 04:11:44.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.625105 7103 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [term 4 FOLLOWER]: Refusing update from remote peer acd494c7bf49499fab7dec85b5d9b980: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch) 04:11:44.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.629159 7151 consensus_queue.cc:984] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s 04:11:44.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.639535 7154 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acd494c7bf49499fab7dec85b5d9b980. Latest consensus state: current_term: 4 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:44.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.640369 7154 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:44.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.657729 7151 sys_catalog.cc:339] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:44.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.658442 7151 sys_catalog.cc:342] T 00000000000000000000000000000000 P acd494c7bf49499fab7dec85b5d9b980 [sys.catalog]: This master's current role is: LEADER 04:11:44.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.659353 7154 sys_catalog.cc:339] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "acd494c7bf49499fab7dec85b5d9b980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f298b835d8664427a9eca376e52d20f9" member_type: VOTER last_known_addr { host: "127.6.32.190" port: 43097 } } peers { permanent_uuid: "acd494c7bf49499fab7dec85b5d9b980" member_type: VOTER last_known_addr { host: "127.6.32.189" port: 43901 } } peers { permanent_uuid: "6713f258192f4403a1e3c5d663ca5280" member_type: VOTER last_known_addr { host: "127.6.32.188" port: 36321 } } } 04:11:44.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.660279 7154 sys_catalog.cc:342] T 00000000000000000000000000000000 P 6713f258192f4403a1e3c5d663ca5280 [sys.catalog]: This master's current role is: FOLLOWER 04:11:44.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:44.740324 7127 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.821s user 0.275s sys 0.420s 04:11:44.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:44.740603 7127 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.821s user 0.275s sys 0.420s 04:11:44.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.784138 7127 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:44.935 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.935489 7127 rpc_server.cc:205] RPC server started. Bound to: 127.6.32.190:43097 04:11:44.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.939529 7127 webserver.cc:374] Webserver started at http://127.6.32.190:36831/ using document root and password file 04:11:44.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.940490 7127 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster6256948055019097247/master-0/data/info.pb 04:11:44.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.942565 6274 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 7127 04:11:44.943 [DEBUG - Time-limited test] (MiniKuduCluster.java:165) Response: 04:11:44.960 [DEBUG - New I/O worker #122] (Negotiator.java:291) peer /127.6.32.189:43901 sent connection negotiation error: Not authorized: token expired 04:11:44.961 [DEBUG - New I/O worker #122] (Connection.java:691) [peer master-127.6.32.189:43901(127.6.32.189:43901)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 04:11:44.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.961779 7208 sys_catalog.cc:188] Verifying existing consensus state 04:11:44.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.967603 7208 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P f298b835d8664427a9eca376e52d20f9: Bootstrap starting. 04:11:44.973 [DEBUG - New I/O worker #123] (Connection.java:691) [peer master-127.6.32.189:43901(127.6.32.189:43901)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because this connection will be used to acquire a new token and therefore requires primary credentials 04:11:44.973 [INFO - New I/O worker #123] (ConnectToCluster.java:361) Unable to connect to master 127.6.32.189:43901: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because this connection will be used to acquire a new token and therefore requires primary credentials 04:11:44.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:44.973908 7209 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45168: token expired 04:11:44.975 [DEBUG - New I/O worker #124] (Connection.java:691) [peer master-127.6.32.188:36321(127.6.32.188:36321)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because this connection will be used to acquire a new token and therefore requires primary credentials 04:11:44.975 [INFO - New I/O worker #124] (ConnectToCluster.java:361) Unable to connect to master 127.6.32.188:36321: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because this connection will be used to acquire a new token and therefore requires primary credentials 04:11:44.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:44.977525 7211 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:44.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.962794 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53276 04:11:44.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.974986 (+ 12192us) server_negotiation.cc:184] Beginning negotiation 04:11:44.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.975005 (+ 19us) server_negotiation.cc:373] Waiting for connection header 04:11:44.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.975037 (+ 32us) server_negotiation.cc:381] Connection header received 04:11:44.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.975175 (+ 138us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:44.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.975189 (+ 14us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:44.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.975309 (+ 120us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:44.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.975468 (+ 159us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:44.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.976513 (+ 1045us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53276: BlockingRecv error: recv got EOF from 127.0.0.1:53276 (error 108) 04:11:44.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":11890,"thread_start_us":8584,"threads_started":1} 04:11:44.979 [DEBUG - New I/O worker #125] (Connection.java:691) [peer master-127.6.32.190:43097(127.6.32.190:43097)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because this connection will be used to acquire a new token and therefore requires primary credentials 04:11:44.979 [INFO - New I/O worker #125] (ConnectToCluster.java:361) Unable to connect to master 127.6.32.190:43097: server requires authentication, but client Kerberos credentials (TGT) have expired. Authentication tokens were not used because this connection will be used to acquire a new token and therefore requires primary credentials 04:11:44.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:44.980315 7212 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:44.981 [DEBUG - Time-limited test] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:11:44.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.962621 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:45172 04:11:44.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.972243 (+ 9622us) server_negotiation.cc:184] Beginning negotiation 04:11:44.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.972258 (+ 15us) server_negotiation.cc:373] Waiting for connection header 04:11:44.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.972289 (+ 31us) server_negotiation.cc:381] Connection header received 04:11:44.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.972378 (+ 89us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:44.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.972391 (+ 13us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:44.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.972506 (+ 115us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:44.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.972673 (+ 167us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:44.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.979383 (+ 6710us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45172: BlockingRecv error: recv got EOF from 127.0.0.1:45172 (error 108) 04:11:44.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":9395,"thread_start_us":10828,"threads_started":1} 04:11:44.983 [DEBUG - main] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:11:44.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:44.986001 7213 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:44.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.966224 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:60540 04:11:44.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.974370 (+ 8146us) server_negotiation.cc:184] Beginning negotiation 04:11:44.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.974393 (+ 23us) server_negotiation.cc:373] Waiting for connection header 04:11:44.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.974433 (+ 40us) server_negotiation.cc:381] Connection header received 04:11:44.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.979078 (+ 4645us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:44.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.979104 (+ 26us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:44.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.979215 (+ 111us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:44.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.982292 (+ 3077us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:44.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.984100 (+ 1808us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60540: BlockingRecv error: recv got EOF from 127.0.0.1:60540 (error 108) 04:11:44.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":7671,"thread_start_us":7065,"threads_started":1} 04:11:44.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:44 dist-test-slave-dist-test-slave-fvsv krb5kdc[6283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608701, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.32.189@KRBTEST.COM for kudu/127.6.32.190@KRBTEST.COM 04:11:44.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:44.997190 6274 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 7127 04:11:45.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:45.021695 7152 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.945775 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.6.32.190:43097 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.946245 (+ 470us) negotiation.cc:98] Waiting for socket to connect 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.946269 (+ 24us) client_negotiation.cc:168] Beginning negotiation 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.946404 (+ 135us) client_negotiation.cc:245] Sending NEGOTIATE NegotiatePB request 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.979140 (+ 32736us) client_negotiation.cc:262] Received NEGOTIATE NegotiatePB response 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.979172 (+ 32us) client_negotiation.cc:356] Received NEGOTIATE response from server 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.980187 (+ 1015us) client_negotiation.cc:183] Negotiated authn=SASL 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.980569 (+ 382us) client_negotiation.cc:472] Sending TLS_HANDSHAKE message to server 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.980613 (+ 44us) client_negotiation.cc:245] Sending TLS_HANDSHAKE NegotiatePB request 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.988669 (+ 8056us) client_negotiation.cc:262] Received TLS_HANDSHAKE NegotiatePB response 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.988696 (+ 27us) client_negotiation.cc:485] Received TLS_HANDSHAKE response from server 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.989535 (+ 839us) client_negotiation.cc:472] Sending TLS_HANDSHAKE message to server 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.989585 (+ 50us) client_negotiation.cc:245] Sending TLS_HANDSHAKE NegotiatePB request 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.992542 (+ 2957us) client_negotiation.cc:262] Received TLS_HANDSHAKE NegotiatePB response 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.992624 (+ 82us) client_negotiation.cc:485] Received TLS_HANDSHAKE response from server 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.992901 (+ 277us) client_negotiation.cc:507] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.993258 (+ 357us) client_negotiation.cc:562] Initiating SASL GSSAPI handshake 04:11:45.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.993272 (+ 14us) client_negotiation.cc:587] Calling sasl_client_start() 04:11:45.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:44.998279 (+ 5007us) client_negotiation.cc:245] Sending SASL_INITIATE NegotiatePB request 04:11:45.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:45.020593 (+ 22314us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.32.190:43097: BlockingRecv error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) 04:11:45.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"client-negotiator.queue_time_us":266} 04:11:45.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:45.030045 6274 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 6997 04:11:45.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:45.055343 6274 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 7060 04:11:46.099 [INFO - main] (RetryRule.java:136) Creating RetryStatement with result reporter and retry count of 0 (no retries) .04:11:46.100 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskusvpEt/test-tmp 04:11:46.100 [INFO - main] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin 04:11:46.100 [INFO - main] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskusvpEt/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb] 04:11:46.101 [DEBUG - main] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 0 enable_kerberos: true cluster_root: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434" extra_master_flags: "--authn_token_validity_seconds=10" hms_mode: NONE mini_kdc_options { ticket_lifetime: "10s" renew_lifetime: "20s" } } 04:11:46.280 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:11:46.280 [DEBUG - main] (MiniKuduCluster.java:155) Request: start_cluster { } 04:11:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Loading random data 04:11:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Initializing database '/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/principal' for realm 'KRBTEST.COM', 04:11:46.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) master key name 'K/M@KRBTEST.COM' 04:11:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:46 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): setting up network... 04:11:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:46 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) 04:11:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked 04:11:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 04:11:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:46 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): skipping unrecognized local address family 17 04:11:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:46 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): skipping unrecognized local address family 17 04:11:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:46 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): set up 1 sockets 04:11:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:46 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): commencing operation 04:11:46.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: starting... 04:11:46.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:46.868196 7217 mini_kdc.cc:119] Time spent starting KDC: real 0.560s user 0.001s sys 0.016s 04:11:46.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy 04:11:46.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:46.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-admin@KRBTEST.COM" created. 04:11:46.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy 04:11:46.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:46.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-user@KRBTEST.COM" created. 04:11:46.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy 04:11:46.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:46.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "joe-interloper@KRBTEST.COM" created. 04:11:46.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 04:11:46.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/test-user.keytab. 04:11:46.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/test-user.keytab. 04:11:46.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/test-user.keytab. 04:11:46.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/test-user.keytab. 04:11:46.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:46 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:46.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:11:47.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.7.12.126@KRBTEST.COM; defaulting to no policy 04:11:47.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:47.007 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.7.12.126@KRBTEST.COM" created. 04:11:47.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:47.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.7.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.126.keytab. 04:11:47.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.7.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.126.keytab. 04:11:47.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.7.12.126 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.126.keytab. 04:11:47.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.7.12.126 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.126.keytab. 04:11:47.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.028592 7217 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:47.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:47.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/wal 04:11:47.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/data 04:11:47.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:47.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:47.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.7.12.126:43389 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.7.12.126 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/logs 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/data/info.pb 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.126.keytab 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.7.12.126 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:47.029 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.7.12.126:43389,127.7.12.125:41989,127.7.12.124:41359 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:47.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:47.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:47.173808 7237 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:47.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:47.174185 7237 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.7.12.126 04:11:47.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:47.174510 7237 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:47.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:47.185516 7237 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:47.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:47.185670 7237 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:47.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:47.185813 7237 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:47.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:47.185942 7237 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:47.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:47.186125 7237 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:47.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:47.186300 7237 flags.cc:404] Enabled experimental flag: --principal=kudu/127.7.12.126 04:11:47.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.204704 7237 master_main.cc:98] Master server non-default flags: 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/data 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/wal 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.7.12.126:43389,127.7.12.125:41989,127.7.12.124:41359 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.7.12.126:43389 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.126.keytab 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.7.12.126 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/data/info.pb 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.7.12.126 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/logs 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:47.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:47.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:47.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:47.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:47.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:47.219 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.218911 7237 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 190516us 04:11:47.220 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:47 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608707, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:47.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.221303 7237 init.cc:359] Logged in from keytab as kudu/127.7.12.126@KRBTEST.COM (short username kudu) 04:11:47.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.224647 7237 fs_manager.cc:263] Metadata directory not provided 04:11:47.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.224772 7237 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:47.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.225006 7237 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:47.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.225106 7237 server_base.cc:449] Attempting to create new FS layout instead 04:11:47.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.228096 7237 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/data/instance: 04:11:47.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1bfb2b1ddd09416391ffc0f9664ef739" 04:11:47.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:47 on dist-test-slave-dist-test-slave-fvsv" 04:11:47.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.228808 7237 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/wal/instance: 04:11:47.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1bfb2b1ddd09416391ffc0f9664ef739" 04:11:47.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:47 on dist-test-slave-dist-test-slave-fvsv" 04:11:47.237 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.237625 7237 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s 04:11:47.238 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.238818 7237 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:47.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.239370 7237 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:47.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.246321 7237 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:11:47.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.246443 7237 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/wal 04:11:47.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "1bfb2b1ddd09416391ffc0f9664ef739" 04:11:47.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:47 on dist-test-slave-dist-test-slave-fvsv" 04:11:47.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.246654 7237 fs_report.cc:352] FS layout report 04:11:47.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:47.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/wal 04:11:47.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/wal 04:11:47.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/data/data 04:11:47.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:47.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:47.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:47.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:47.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:47.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:47.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:47.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:47.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:47.247 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:47.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:47.891615 7237 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.608s user 0.000s sys 0.002s 04:11:47.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:47.891863 7237 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.609s user 0.000s sys 0.002s 04:11:47.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.920859 7237 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:47.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.992184 7237 rpc_server.cc:205] RPC server started. Bound to: 127.7.12.126:43389 04:11:47.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.995455 7237 webserver.cc:374] Webserver started at http://127.7.12.126:40173/ using document root and password file 04:11:47.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:47.996608 7237 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-0/data/info.pb 04:11:48.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.000311 7299 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:48.002 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.001888 7217 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 7237 04:11:48.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.7.12.125@KRBTEST.COM; defaulting to no policy 04:11:48.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:48.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.7.12.125@KRBTEST.COM" created. 04:11:48.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.018330 7299 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } has no permanent_uuid. Determining permanent_uuid... 04:11:48.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:48 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608707, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.12.126@KRBTEST.COM for kudu/127.7.12.126@KRBTEST.COM 04:11:48.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:48.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.7.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.125.keytab. 04:11:48.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.7.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.125.keytab. 04:11:48.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.7.12.125 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.125.keytab. 04:11:48.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.7.12.125 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.125.keytab. 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.042150 7217 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/wal 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/data 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.7.12.125:41989 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.7.12.125 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:48.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/logs 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/data/info.pb 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.125.keytab 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.7.12.125 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:48.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.7.12.126:43389,127.7.12.125:41989,127.7.12.124:41359 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:48.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.049211 7299 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } has no permanent_uuid. Determining permanent_uuid... 04:11:48.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.050906 7299 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.125:41989: Network error: Client connection negotiation failed: client connection to 127.7.12.125:41989: connect: Connection refused (error 111) 04:11:48.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.070307 7299 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } attempt: 1 04:11:48.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.072017 7299 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.125:41989: Network error: Client connection negotiation failed: client connection to 127.7.12.125:41989: connect: Connection refused (error 111) 04:11:48.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.120363 7299 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } attempt: 2 04:11:48.122 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.122073 7299 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.125:41989: Network error: Client connection negotiation failed: client connection to 127.7.12.125:41989: connect: Connection refused (error 111) 04:11:48.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:48.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.193862 7304 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:48.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.194216 7304 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.7.12.125 04:11:48.194 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.194486 7304 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:48.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.205592 7304 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:48.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.205734 7304 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:48.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.205842 7304 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:48.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.205930 7304 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:48.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.206048 7304 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:48.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.206166 7304 flags.cc:404] Enabled experimental flag: --principal=kudu/127.7.12.125 04:11:48.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.225580 7304 master_main.cc:98] Master server non-default flags: 04:11:48.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/data 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/wal 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.7.12.126:43389,127.7.12.125:41989,127.7.12.124:41359 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.7.12.125:41989 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.125.keytab 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.7.12.125 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/data/info.pb 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.7.12.125 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/logs 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:48.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:48.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:48.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.233422 7299 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } attempt: 3 04:11:48.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.235008 7299 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.125:41989: Network error: Client connection negotiation failed: client connection to 127.7.12.125:41989: connect: Connection refused (error 111) 04:11:48.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.240576 7304 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 191016us 04:11:48.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:48 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608708, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:48.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.242828 7304 init.cc:359] Logged in from keytab as kudu/127.7.12.125@KRBTEST.COM (short username kudu) 04:11:48.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.245642 7304 fs_manager.cc:263] Metadata directory not provided 04:11:48.245 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.245759 7304 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:48.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.245992 7304 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:48.246 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.246091 7304 server_base.cc:449] Attempting to create new FS layout instead 04:11:48.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.249073 7304 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/data/instance: 04:11:48.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" 04:11:48.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:48 on dist-test-slave-dist-test-slave-fvsv" 04:11:48.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.249764 7304 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/wal/instance: 04:11:48.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" 04:11:48.249 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:48 on dist-test-slave-dist-test-slave-fvsv" 04:11:48.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.258698 7304 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s 04:11:48.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.259944 7304 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:48.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.260452 7304 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:48.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.267515 7304 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:11:48.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.267637 7304 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/wal 04:11:48.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" 04:11:48.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:48 on dist-test-slave-dist-test-slave-fvsv" 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.267843 7304 fs_report.cc:352] FS layout report 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/wal 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/wal 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/data/data 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:48.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:48.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.411397 7299 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } attempt: 4 04:11:48.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.413893 7299 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.125:41989: Network error: Client connection negotiation failed: client connection to 127.7.12.125:41989: connect: Connection refused (error 111) 04:11:48.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.717298 7299 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } attempt: 5 04:11:48.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.719826 7299 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.125:41989: Network error: Client connection negotiation failed: client connection to 127.7.12.125:41989: connect: Connection refused (error 111) 04:11:48.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.929122 7304 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.634s user 0.000s sys 0.002s 04:11:48.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:48.929339 7304 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.634s user 0.000s sys 0.002s 04:11:48.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:48.960878 7304 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:49.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.040113 7304 rpc_server.cc:205] RPC server started. Bound to: 127.7.12.125:41989 04:11:49.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.042939 7304 webserver.cc:374] Webserver started at http://127.7.12.125:33313/ using document root and password file 04:11:49.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.043850 7304 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-1/data/info.pb 04:11:49.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.047240 7217 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 7304 04:11:49.047 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.047439 7366 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:49.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.7.12.124@KRBTEST.COM; defaulting to no policy 04:11:49.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.067556 7366 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } has no permanent_uuid. Determining permanent_uuid... 04:11:49.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:49.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.7.12.124@KRBTEST.COM" created. 04:11:49.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:49 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608708, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.12.125@KRBTEST.COM for kudu/127.7.12.126@KRBTEST.COM 04:11:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 04:11:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.7.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.124.keytab. 04:11:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.7.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.124.keytab. 04:11:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.7.12.124 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.124.keytab. 04:11:49.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.7.12.124 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.124.keytab. 04:11:49.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.100664 7217 external_mini_cluster.cc:938] Running /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/wal 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/data 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.7.12.124:41359 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.7.12.124 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/logs 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/data/info.pb 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:49.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 04:11:49.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 04:11:49.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 04:11:49.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.124.keytab 04:11:49.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.7.12.124 04:11:49.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:49.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:49.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:49.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:49.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:49.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.7.12.126:43389,127.7.12.125:41989,127.7.12.124:41359 with env {KRB5CCNAME=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 04:11:49.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.103572 7366 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } has no permanent_uuid. Determining permanent_uuid... 04:11:49.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:49 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608708, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.12.125@KRBTEST.COM for kudu/127.7.12.125@KRBTEST.COM 04:11:49.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.126260 7366 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } has no permanent_uuid. Determining permanent_uuid... 04:11:49.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.127977 7366 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.124:41359: Network error: Client connection negotiation failed: client connection to 127.7.12.124:41359: connect: Connection refused (error 111) 04:11:49.147 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.147348 7366 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } attempt: 1 04:11:49.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.148946 7366 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.124:41359: Network error: Client connection negotiation failed: client connection to 127.7.12.124:41359: connect: Connection refused (error 111) 04:11:49.197 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.197222 7366 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } attempt: 2 04:11:49.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.198856 7366 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.124:41359: Network error: Client connection negotiation failed: client connection to 127.7.12.124:41359: connect: Connection refused (error 111) 04:11:49.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 04:11:49.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.253223 7372 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 04:11:49.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.253587 7372 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.7.12.124 04:11:49.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.253815 7372 flags.cc:404] Enabled unsafe flag: --never_fsync=true 04:11:49.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.254221 7299 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } attempt: 6 04:11:49.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:49 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608707, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.12.126@KRBTEST.COM for kudu/127.7.12.125@KRBTEST.COM 04:11:49.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.268517 7372 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=10 04:11:49.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.268714 7372 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 04:11:49.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.268878 7372 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 04:11:49.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.269013 7372 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 04:11:49.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.269134 7372 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 04:11:49.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.269260 7372 flags.cc:404] Enabled experimental flag: --principal=kudu/127.7.12.124 04:11:49.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.274572 7299 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } has no permanent_uuid. Determining permanent_uuid... 04:11:49.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.276212 7299 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.124:41359: Network error: Client connection negotiation failed: client connection to 127.7.12.124:41359: connect: Connection refused (error 111) 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.289963 7372 master_main.cc:98] Master server non-default flags: 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/data 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/wal 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=10 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.7.12.126:43389,127.7.12.125:41989,127.7.12.124:41359 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.7.12.124:41359 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kudu_127.7.12.124.keytab 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.7.12.124 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/data/info.pb 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.7.12.124 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 04:11:49.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/logs 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.11.0-cdh6.x-SNAPSHOT 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 9fd07afe29a9473508cb10cabf49e752bc2d2909 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type FASTDEBUG 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 26 Jun 2019 20:30:26 PST on quasar-tfmyxo-1.vpc.cloudera.com 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 374348 04:11:49.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) TSAN enabled 04:11:49.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.304086 7372 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 191516us 04:11:49.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:49 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:49.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.306401 7372 init.cc:359] Logged in from keytab as kudu/127.7.12.124@KRBTEST.COM (short username kudu) 04:11:49.308 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.308507 7299 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } attempt: 1 04:11:49.309 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.309764 7372 fs_manager.cc:263] Metadata directory not provided 04:11:49.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.309952 7372 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 04:11:49.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.310000 7299 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.124:41359: Network error: Client connection negotiation failed: client connection to 127.7.12.124:41359: connect: Connection refused (error 111) 04:11:49.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.310317 7372 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file 04:11:49.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.310217 7366 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } attempt: 3 04:11:49.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.310441 7372 server_base.cc:449] Attempting to create new FS layout instead 04:11:49.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.311640 7366 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.124:41359: Network error: Client connection negotiation failed: client connection to 127.7.12.124:41359: connect: Connection refused (error 111) 04:11:49.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.313616 7372 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/data/instance: 04:11:49.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "dff5e6e71a684c1db979b7651f240672" 04:11:49.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:49 on dist-test-slave-dist-test-slave-fvsv" 04:11:49.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.314328 7372 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/wal/instance: 04:11:49.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "dff5e6e71a684c1db979b7651f240672" 04:11:49.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:49 on dist-test-slave-dist-test-slave-fvsv" 04:11:49.323 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.323227 7372 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.008s sys 0.003s 04:11:49.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.324457 7372 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 04:11:49.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.324942 7372 file_cache.cc:465] Constructed file cache lbm with capacity 419430 04:11:49.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.331787 7372 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.331909 7372 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/data,/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/wal 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "dff5e6e71a684c1db979b7651f240672" 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-06-27 04:11:49 on dist-test-slave-dist-test-slave-fvsv" 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.332111 7372 fs_report.cc:352] FS layout report 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/wal 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/wal 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/data/data 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 04:11:49.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 04:11:49.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.378295 7299 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } attempt: 2 04:11:49.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.380832 7299 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.124:41359: Network error: Client connection negotiation failed: client connection to 127.7.12.124:41359: connect: Connection refused (error 111) 04:11:49.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.457129 7299 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } attempt: 3 04:11:49.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.458791 7299 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.124:41359: Network error: Client connection negotiation failed: client connection to 127.7.12.124:41359: connect: Connection refused (error 111) 04:11:49.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.487994 7366 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } attempt: 4 04:11:49.489 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.489588 7366 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.124:41359: Network error: Client connection negotiation failed: client connection to 127.7.12.124:41359: connect: Connection refused (error 111) 04:11:49.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.596084 7299 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } attempt: 4 04:11:49.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.597791 7299 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.124:41359: Network error: Client connection negotiation failed: client connection to 127.7.12.124:41359: connect: Connection refused (error 111) 04:11:49.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.792881 7366 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } attempt: 5 04:11:49.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.794440 7366 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.124:41359: Network error: Client connection negotiation failed: client connection to 127.7.12.124:41359: connect: Connection refused (error 111) 04:11:49.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:49.881111 7299 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } attempt: 5 04:11:49.883 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:49.882913 7299 consensus_peers.cc:607] Error getting permanent uuid from config peer 127.7.12.124:41359: Network error: Client connection negotiation failed: client connection to 127.7.12.124:41359: connect: Connection refused (error 111) 04:11:50.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:50.206192 7372 thread.cc:599] rpc reactor (reactor) Time spent creating pthread: real 0.849s user 0.296s sys 0.551s 04:11:50.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:50.206387 7372 thread.cc:566] rpc reactor (reactor) Time spent starting thread: real 0.849s user 0.296s sys 0.551s 04:11:50.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.239296 7372 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 04:11:50.312 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.312268 7372 rpc_server.cc:205] RPC server started. Bound to: 127.7.12.124:41359 04:11:50.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.315582 7372 webserver.cc:374] Webserver started at http://127.7.12.124:44221/ using document root and password file 04:11:50.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.316769 7372 server_base.cc:629] Dumped server information to /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/master-2/data/info.pb 04:11:50.319 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.318534 7217 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master as pid 7372 04:11:50.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.321262 7435 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 04:11:50.329 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.328770 7366 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } attempt: 6 04:11:50.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.338726 7435 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } has no permanent_uuid. Determining permanent_uuid... 04:11:50.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608708, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.12.125@KRBTEST.COM for kudu/127.7.12.124@KRBTEST.COM 04:11:50.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.12.124@KRBTEST.COM 04:11:50.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.12.124@KRBTEST.COM for kudu/127.7.12.126@KRBTEST.COM 04:11:50.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.376519 7366 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc: Bootstrap starting. 04:11:50.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.377583 7435 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } has no permanent_uuid. Determining permanent_uuid... 04:11:50.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.380937 7366 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc: No blocks or log segments found. Creating new log. 04:11:50.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.383373 7366 log.cc:525] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc: Log is configured to *not* fsync() on all Append() calls 04:11:50.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.12.125@KRBTEST.COM 04:11:50.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.387758 7366 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc: No bootstrap required, opened a new log 04:11:50.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.12.124@KRBTEST.COM for kudu/127.7.12.125@KRBTEST.COM 04:11:50.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.395228 7299 consensus_peers.cc:617] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } attempt: 6 04:11:50.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.402709 7435 sys_catalog.cc:306] member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } has no permanent_uuid. Determining permanent_uuid... 04:11:50.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.405596 7366 raft_consensus.cc:340] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } 04:11:50.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.406332 7366 raft_consensus.cc:366] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:50.406 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.406563 7366 raft_consensus.cc:705] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb56e7b2bd8745688ceafb0cf0a278fc, State: Initialized, Role: FOLLOWER 04:11:50.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.407284 7366 consensus_queue.cc:229] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } 04:11:50.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.410573 7366 sys_catalog.cc:339] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608709, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.12.124@KRBTEST.COM for kudu/127.7.12.124@KRBTEST.COM 04:11:50.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.12.126@KRBTEST.COM 04:11:50.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608707, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.12.126@KRBTEST.COM for kudu/127.7.12.124@KRBTEST.COM 04:11:50.422 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.411399 7366 sys_catalog.cc:342] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [sys.catalog]: This master's current role is: FOLLOWER 04:11:50.424 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.423704 7366 sys_catalog.cc:427] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [sys.catalog]: configured and running, proceeding with master startup. 04:11:50.428 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:11:50.429 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_kdc_env_vars { } 04:11:50.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.428632 7217 external_mini_cluster.cc:628] 0 TS(s) registered with all masters 04:11:50.431 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_kdc_env_vars { env_vars { key: "KUDU_ENABLE_KRB5_REALM_FIX" value: "yes" } env_vars { key: "KRB5_KDC_PROFILE" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/kdc.conf" } env_vars { key: "KRB5_CONFIG" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/krb5.conf" } env_vars { key: "KRB5CCNAME" value: "/tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/krb5cc" } } 04:11:50.432 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_masters { } 04:11:50.433 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.7.12.126" port: 43389 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.7.12.125" port: 41989 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.7.12.124" port: 41359 } } } 04:11:50.433 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_tservers { } 04:11:50.435 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:11:50.435 [INFO - main] (MiniKuduCluster.java:439) Running kinit for user test-admin 04:11:50.435 [DEBUG - main] (MiniKuduCluster.java:155) Request: kinit { username: "test-admin" } 04:11:50.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.437618 7435 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672: Bootstrap starting. 04:11:50.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.441273 7435 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672: No blocks or log segments found. Creating new log. 04:11:50.441 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.441817 7299 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739: Bootstrap starting. 04:11:50.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.445416 7299 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739: No blocks or log segments found. Creating new log. 04:11:50.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.447301 7435 log.cc:525] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672: Log is configured to *not* fsync() on all Append() calls 04:11:50.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.447453 7299 log.cc:525] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739: Log is configured to *not* fsync() on all Append() calls 04:11:50.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.450605 7299 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739: No bootstrap required, opened a new log 04:11:50.452 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.452340 7435 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672: No bootstrap required, opened a new log 04:11:50.472 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.472162 7435 raft_consensus.cc:340] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } 04:11:50.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.473361 7435 raft_consensus.cc:366] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:50.473 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.473728 7435 raft_consensus.cc:705] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff5e6e71a684c1db979b7651f240672, State: Initialized, Role: FOLLOWER 04:11:50.475 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.474579 7435 consensus_queue.cc:229] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } 04:11:50.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 04:11:50.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.479162 7435 sys_catalog.cc:339] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.479900 7435 sys_catalog.cc:342] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [sys.catalog]: This master's current role is: FOLLOWER 04:11:50.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.481703 7435 sys_catalog.cc:427] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [sys.catalog]: configured and running, proceeding with master startup. 04:11:50.486 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 04:11:50.494 [DEBUG - main] (MiniKuduCluster.java:165) Response: 04:11:50.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.481549 7299 raft_consensus.cc:340] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } 04:11:50.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.502768 7299 raft_consensus.cc:366] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 04:11:50.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.503531 7299 raft_consensus.cc:705] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfb2b1ddd09416391ffc0f9664ef739, State: Initialized, Role: FOLLOWER 04:11:50.505 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.504647 7299 consensus_queue.cc:229] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } 04:11:50.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.508057 7299 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.508808 7299 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [sys.catalog]: This master's current role is: FOLLOWER 04:11:50.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.510164 7299 sys_catalog.cc:427] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [sys.catalog]: configured and running, proceeding with master startup. 04:11:50.511 [DEBUG - main] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/krb5cc 04:11:50.513 [DEBUG - main] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:11:50.528 [DEBUG - New I/O worker #127] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:11:50.529 [DEBUG - New I/O worker #127] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/krb5cc 04:11:50.530 [DEBUG - New I/O worker #127] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:11:50.530 [INFO - New I/O worker #127] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:11:50.531 [DEBUG - New I/O worker #127] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.7.12.126 04:11:50.545 [DEBUG - New I/O worker #128] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:11:50.546 [DEBUG - New I/O worker #128] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/krb5cc 04:11:50.547 [DEBUG - New I/O worker #128] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:11:50.547 [INFO - New I/O worker #128] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:11:50.548 [DEBUG - New I/O worker #128] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.7.12.124 04:11:50.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.12.124@KRBTEST.COM 04:11:50.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.12.126@KRBTEST.COM 04:11:50.566 [DEBUG - New I/O worker #129] (SecurityContext.java:217) Refreshing Kerberos credentials... 04:11:50.569 [DEBUG - New I/O worker #129] (SecurityUtil.java:104) Using ticketCache: /tmp/dist-test-taskusvpEt/test-tmp/mini-kudu-cluster983779746327727434/krb5kdc/krb5cc 04:11:50.570 [DEBUG - New I/O worker #129] (SecurityUtil.java:118) Logged in as subject: test-admin@KRBTEST.COM 04:11:50.608 [INFO - New I/O worker #129] (SecurityContext.java:254) Successfully refreshed Kerberos credentials from ticket cache 04:11:50.609 [DEBUG - New I/O worker #129] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.7.12.125 04:11:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.508203 7457 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.614852 7457 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [sys.catalog]: This master's current role is: FOLLOWER 04:11:50.619 [DEBUG - New I/O worker #127] (Negotiator.java:764) Authenticated connection [id: 0x4446307c, /127.0.0.1:57792 => /127.7.12.126:43389] using SASL/GSSAPI 04:11:50.629 [DEBUG - New I/O worker #128] (Negotiator.java:764) Authenticated connection [id: 0xefed4dbf, /127.0.0.1:40354 => /127.7.12.124:41359] using SASL/GSSAPI 04:11:50.635 [INFO - New I/O worker #128] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: Catalog manager is not initialized. State: Starting 04:11:50.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Jun 27 04:11:50 dist-test-slave-dist-test-slave-fvsv krb5kdc[7226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1561608710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.12.125@KRBTEST.COM 04:11:50.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:50.642630 7464 catalog_manager.cc:1122] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:11:50.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:50.659190 7464 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:11:50.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.410796 7446 sys_catalog.cc:339] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.665 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.665480 7446 sys_catalog.cc:342] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [sys.catalog]: This master's current role is: FOLLOWER 04:11:50.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.665935 7446 raft_consensus.cc:472] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 04:11:50.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.666339 7446 raft_consensus.cc:494] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } 04:11:50.667 [DEBUG - New I/O worker #129] (Negotiator.java:764) Authenticated connection [id: 0xe5d7aaf0, /127.0.0.1:59120 => /127.7.12.125:41989] using SASL/GSSAPI 04:11:50.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.668936 7446 leader_election.cc:296] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bfb2b1ddd09416391ffc0f9664ef739 (127.7.12.126:43389), dff5e6e71a684c1db979b7651f240672 (127.7.12.124:41359) 04:11:50.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.670182 7276 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" is_pre_election: true 04:11:50.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.671034 7276 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb56e7b2bd8745688ceafb0cf0a278fc in term 0. 04:11:50.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.672422 7315 leader_election.cc:310] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bfb2b1ddd09416391ffc0f9664ef739, eb56e7b2bd8745688ceafb0cf0a278fc; no voters: 04:11:50.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.673296 7446 raft_consensus.cc:2644] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [term 0 FOLLOWER]: Leader pre-election won for term 1 04:11:50.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.673481 7446 raft_consensus.cc:472] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 04:11:50.673 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.673686 7446 raft_consensus.cc:2890] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [term 0 FOLLOWER]: Advancing to term 1 04:11:50.677 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.677135 7446 raft_consensus.cc:494] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } 04:11:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:50.678086 7255 tablet.cc:1799] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:50.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.678300 7446 leader_election.cc:296] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [CANDIDATE]: Term 1 election: Requested vote from peers 1bfb2b1ddd09416391ffc0f9664ef739 (127.7.12.126:43389), dff5e6e71a684c1db979b7651f240672 (127.7.12.124:41359) 04:11:50.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.680039 7276 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" 04:11:50.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.680514 7276 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [term 0 FOLLOWER]: Advancing to term 1 04:11:50.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.685783 7276 raft_consensus.cc:2350] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb56e7b2bd8745688ceafb0cf0a278fc in term 1. 04:11:50.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.686878 7315 leader_election.cc:310] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bfb2b1ddd09416391ffc0f9664ef739, eb56e7b2bd8745688ceafb0cf0a278fc; no voters: 04:11:50.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.687801 7446 raft_consensus.cc:2644] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [term 1 FOLLOWER]: Leader election won for term 1 04:11:50.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.688257 7446 raft_consensus.cc:667] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [term 1 LEADER]: Becoming Leader. State: Replica: eb56e7b2bd8745688ceafb0cf0a278fc, State: Running, Role: LEADER 04:11:50.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.688906 7446 consensus_queue.cc:206] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } 04:11:50.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.480443 7452 sys_catalog.cc:339] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.701220 7452 sys_catalog.cc:342] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [sys.catalog]: This master's current role is: FOLLOWER 04:11:50.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:50.467600 7322 tablet.cc:1799] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:50.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.744343 7446 sys_catalog.cc:339] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb56e7b2bd8745688ceafb0cf0a278fc. Latest consensus state: current_term: 1 leader_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.745 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.745168 7446 sys_catalog.cc:342] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [sys.catalog]: This master's current role is: LEADER 04:11:50.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.747787 7412 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff5e6e71a684c1db979b7651f240672" is_pre_election: true 04:11:50.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.748627 7412 raft_consensus.cc:2350] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb56e7b2bd8745688ceafb0cf0a278fc in term 0. 04:11:50.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.750003 7412 tablet_service.cc:1288] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff5e6e71a684c1db979b7651f240672" 04:11:50.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.750437 7412 raft_consensus.cc:2890] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [term 0 FOLLOWER]: Advancing to term 1 04:11:50.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:50.753350 7447 catalog_manager.cc:1122] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:11:50.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:50.753630 7447 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:11:50.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.754994 7469 catalog_manager.cc:1064] Loading table and tablet metadata into memory... 04:11:50.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.756242 7412 raft_consensus.cc:2350] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb56e7b2bd8745688ceafb0cf0a278fc in term 1. 04:11:50.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:50.495559 7391 tablet.cc:1799] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672: Can't schedule compaction. Clean time has not been advanced past its initial value. 04:11:50.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.765815 7469 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... 04:11:50.767 [WARN - New I/O worker #129] (ConnectToCluster.java:281) Unable to find the leader master 127.7.12.126:43389,127.7.12.124:41359,127.7.12.125:41989; will retry 04:11:50.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:50.768927 7453 catalog_manager.cc:1122] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672: acquiring CA information for follower catalog manager: Not found: root CA entry not found 04:11:50.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:50.769093 7453 catalog_manager.cc:605] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 04:11:50.788 [WARN - New I/O worker #129] (ConnectToCluster.java:278) None of the provided masters 127.7.12.126:43389,127.7.12.124:41359,127.7.12.125:41989 is a leader; will retry 04:11:50.804 [WARN - New I/O worker #129] (ConnectToCluster.java:278) None of the provided masters 127.7.12.126:43389,127.7.12.124:41359,127.7.12.125:41989 is a leader; will retry 04:11:50.825 [WARN - New I/O worker #129] (ConnectToCluster.java:278) None of the provided masters 127.7.12.126:43389,127.7.12.124:41359,127.7.12.125:41989 is a leader; will retry 04:11:50.843 [WARN - New I/O worker #129] (ConnectToCluster.java:278) None of the provided masters 127.7.12.126:43389,127.7.12.124:41359,127.7.12.125:41989 is a leader; will retry 04:11:50.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.857924 7412 raft_consensus.cc:1184] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [term 1 FOLLOWER]: Refusing update from remote peer eb56e7b2bd8745688ceafb0cf0a278fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:11:50.861 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.860460 7446 consensus_queue.cc:984] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:11:50.862 [WARN - New I/O worker #129] (ConnectToCluster.java:278) None of the provided masters 127.7.12.126:43389,127.7.12.124:41359,127.7.12.125:41989 is a leader; will retry 04:11:50.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.877382 7276 raft_consensus.cc:1184] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [term 1 FOLLOWER]: Refusing update from remote peer eb56e7b2bd8745688ceafb0cf0a278fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 04:11:50.879 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.879256 7446 consensus_queue.cc:984] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 04:11:50.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.899137 7457 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb56e7b2bd8745688ceafb0cf0a278fc. Latest consensus state: current_term: 1 leader_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.899843 7457 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [sys.catalog]: This master's current role is: FOLLOWER 04:11:50.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.900035 7452 sys_catalog.cc:339] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb56e7b2bd8745688ceafb0cf0a278fc. Latest consensus state: current_term: 1 leader_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.900701 7452 sys_catalog.cc:342] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [sys.catalog]: This master's current role is: FOLLOWER 04:11:50.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.907668 7473 mvcc.cc:203] Tried to move safe_time back from 6396349279656288256 to 6396349278989602816. Current Snapshot: MvccSnapshot[committed={T|T < 6396349279656288256}] 04:11:50.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.907919 7446 sys_catalog.cc:339] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.908596 7446 sys_catalog.cc:342] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [sys.catalog]: This master's current role is: LEADER 04:11:50.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.911219 7469 catalog_manager.cc:959] Generated new certificate authority record 04:11:50.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.912147 7452 sys_catalog.cc:339] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.912777 7452 sys_catalog.cc:342] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672 [sys.catalog]: This master's current role is: FOLLOWER 04:11:50.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.913470 7469 catalog_manager.cc:1084] Loading token signing keys... 04:11:50.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.913017 7446 sys_catalog.cc:339] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.919162 7446 sys_catalog.cc:342] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc [sys.catalog]: This master's current role is: LEADER 04:11:50.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.924919 7457 sys_catalog.cc:339] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bfb2b1ddd09416391ffc0f9664ef739" member_type: VOTER last_known_addr { host: "127.7.12.126" port: 43389 } } peers { permanent_uuid: "eb56e7b2bd8745688ceafb0cf0a278fc" member_type: VOTER last_known_addr { host: "127.7.12.125" port: 41989 } } peers { permanent_uuid: "dff5e6e71a684c1db979b7651f240672" member_type: VOTER last_known_addr { host: "127.7.12.124" port: 41359 } } } 04:11:50.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.925627 7457 sys_catalog.cc:342] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739 [sys.catalog]: This master's current role is: FOLLOWER 04:11:50.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:50.946774 7469 catalog_manager.cc:4350] T 00000000000000000000000000000000 P eb56e7b2bd8745688ceafb0cf0a278fc: Generated new TSK 0 04:11:50.951 [WARN - New I/O worker #127] (ConnectToCluster.java:278) None of the provided masters 127.7.12.126:43389,127.7.12.124:41359,127.7.12.125:41989 is a leader; will retry 04:11:51.005 [DEBUG - New I/O worker #129] (AsyncKuduClient.java:2322) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 04:11:51.030 [DEBUG - main] (SecurityUtil.java:121) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication 04:11:51.030 [DEBUG - main] (SecurityContext.java:312) Importing authentication credentials with one authn token, 1 cert(s), and realUser=slave 04:11:51.066 [DEBUG - New I/O worker #137] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.066 [INFO - New I/O worker #137] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.068605 7437 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.037229 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40360 04:11:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.038710 (+ 1481us) server_negotiation.cc:184] Beginning negotiation 04:11:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.038733 (+ 23us) server_negotiation.cc:373] Waiting for connection header 04:11:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.064965 (+ 26232us) server_negotiation.cc:381] Connection header received 04:11:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.065170 (+ 205us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.065194 (+ 24us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.065385 (+ 191us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.065537 (+ 152us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.067428 (+ 1891us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40360: BlockingRecv error: recv got EOF from 127.0.0.1:40360 (error 108) 04:11:51.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":1181} 04:11:51.078 [DEBUG - New I/O worker #136] (Connection.java:691) [peer master-127.7.12.126:43389(127.7.12.126:43389)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.078 [INFO - New I/O worker #136] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.126:43389: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.080582 7440 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.036798 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:57798 04:11:51.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.038345 (+ 1547us) server_negotiation.cc:184] Beginning negotiation 04:11:51.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.038366 (+ 21us) server_negotiation.cc:373] Waiting for connection header 04:11:51.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.077219 (+ 38853us) server_negotiation.cc:381] Connection header received 04:11:51.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.077423 (+ 204us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.077447 (+ 24us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.077641 (+ 194us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.077800 (+ 159us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.079407 (+ 1607us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57798: BlockingRecv error: recv got EOF from 127.0.0.1:57798 (error 108) 04:11:51.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":1198} 04:11:51.090 [DEBUG - New I/O worker #138] (Negotiator.java:764) Authenticated connection [id: 0xcdf9cb81, /127.0.0.1:59126 => /127.7.12.125:41989] using TOKEN/null 04:11:51.147 [DEBUG - New I/O worker #139] (Connection.java:691) [peer master-127.7.12.126:43389(127.7.12.126:43389)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.147 [INFO - New I/O worker #139] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.126:43389: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.148 [DEBUG - New I/O worker #140] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.148 [INFO - New I/O worker #140] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.149194 7440 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.145346 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:57804 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.146220 (+ 874us) server_negotiation.cc:184] Beginning negotiation 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.146252 (+ 32us) server_negotiation.cc:373] Waiting for connection header 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.146300 (+ 48us) server_negotiation.cc:381] Connection header received 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.146439 (+ 139us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.146465 (+ 26us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.146611 (+ 146us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.146766 (+ 155us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.148462 (+ 1696us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57804: BlockingRecv error: recv got EOF from 127.0.0.1:57804 (error 108) 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":559} 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.149843 7437 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.146753 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40366 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.147202 (+ 449us) server_negotiation.cc:184] Beginning negotiation 04:11:51.150 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.147225 (+ 23us) server_negotiation.cc:373] Waiting for connection header 04:11:51.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.147273 (+ 48us) server_negotiation.cc:381] Connection header received 04:11:51.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.147400 (+ 127us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.147422 (+ 22us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.147578 (+ 156us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.147724 (+ 146us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.149128 (+ 1404us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40366: BlockingRecv error: recv got EOF from 127.0.0.1:40366 (error 108) 04:11:51.151 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":144} 04:11:51.208 [DEBUG - New I/O worker #141] (Connection.java:691) [peer master-127.7.12.126:43389(127.7.12.126:43389)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.208 [INFO - New I/O worker #141] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.126:43389: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.210 [DEBUG - New I/O worker #142] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.210 [INFO - New I/O worker #142] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.210069 7440 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.206421 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:57808 04:11:51.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.206851 (+ 430us) server_negotiation.cc:184] Beginning negotiation 04:11:51.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.206873 (+ 22us) server_negotiation.cc:373] Waiting for connection header 04:11:51.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.207259 (+ 386us) server_negotiation.cc:381] Connection header received 04:11:51.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.207416 (+ 157us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.207437 (+ 21us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.207584 (+ 147us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.207728 (+ 144us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.209359 (+ 1631us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57808: BlockingRecv error: recv got EOF from 127.0.0.1:57808 (error 108) 04:11:51.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":124} 04:11:51.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.212576 7437 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.205955 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40370 04:11:51.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.209359 (+ 3404us) server_negotiation.cc:184] Beginning negotiation 04:11:51.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.209385 (+ 26us) server_negotiation.cc:373] Waiting for connection header 04:11:51.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.209425 (+ 40us) server_negotiation.cc:381] Connection header received 04:11:51.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.209548 (+ 123us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.209571 (+ 23us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.209713 (+ 142us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.209860 (+ 147us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.211876 (+ 2016us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40370: BlockingRecv error: recv got EOF from 127.0.0.1:40370 (error 108) 04:11:51.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":3090} 04:11:51.265 [DEBUG - New I/O worker #136] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.265 [INFO - New I/O worker #136] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.266 [DEBUG - New I/O worker #143] (Connection.java:691) [peer master-127.7.12.126:43389(127.7.12.126:43389)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.267 [INFO - New I/O worker #143] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.126:43389: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.268602 7440 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.262835 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:57812 04:11:51.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.263346 (+ 511us) server_negotiation.cc:184] Beginning negotiation 04:11:51.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.263369 (+ 23us) server_negotiation.cc:373] Waiting for connection header 04:11:51.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.264017 (+ 648us) server_negotiation.cc:381] Connection header received 04:11:51.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.266217 (+ 2200us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.266241 (+ 24us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.266389 (+ 148us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.266527 (+ 138us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.267913 (+ 1386us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57812: BlockingRecv error: recv got EOF from 127.0.0.1:57812 (error 108) 04:11:51.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":227} 04:11:51.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.272275 7437 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.264171 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40374 04:11:51.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.264581 (+ 410us) server_negotiation.cc:184] Beginning negotiation 04:11:51.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.264603 (+ 22us) server_negotiation.cc:373] Waiting for connection header 04:11:51.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.264641 (+ 38us) server_negotiation.cc:381] Connection header received 04:11:51.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.264765 (+ 124us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.264787 (+ 22us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.264930 (+ 143us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.265095 (+ 165us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.271575 (+ 6480us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40374: BlockingRecv error: recv got EOF from 127.0.0.1:40374 (error 108) 04:11:51.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":135} 04:11:51.326 [DEBUG - New I/O worker #137] (Connection.java:691) [peer master-127.7.12.126:43389(127.7.12.126:43389)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.326 [INFO - New I/O worker #137] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.126:43389: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.328 [DEBUG - New I/O worker #138] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.328 [INFO - New I/O worker #138] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.330197 7440 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.323305 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:57816 04:11:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.325377 (+ 2072us) server_negotiation.cc:184] Beginning negotiation 04:11:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.325402 (+ 25us) server_negotiation.cc:373] Waiting for connection header 04:11:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.325442 (+ 40us) server_negotiation.cc:381] Connection header received 04:11:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.325572 (+ 130us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.325595 (+ 23us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.325739 (+ 144us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.325900 (+ 161us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.329475 (+ 3575us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57816: BlockingRecv error: recv got EOF from 127.0.0.1:57816 (error 108) 04:11:51.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":1770} 04:11:51.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.331535 7437 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.324103 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40378 04:11:51.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.327686 (+ 3583us) server_negotiation.cc:184] Beginning negotiation 04:11:51.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.327706 (+ 20us) server_negotiation.cc:373] Waiting for connection header 04:11:51.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.327748 (+ 42us) server_negotiation.cc:381] Connection header received 04:11:51.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.327877 (+ 129us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.327900 (+ 23us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.328045 (+ 145us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.328192 (+ 147us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.330802 (+ 2610us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40378: BlockingRecv error: recv got EOF from 127.0.0.1:40378 (error 108) 04:11:51.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":3293} 04:11:51.385 [DEBUG - New I/O worker #139] (Connection.java:691) [peer master-127.7.12.126:43389(127.7.12.126:43389)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.386 [INFO - New I/O worker #139] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.126:43389: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.387 [DEBUG - New I/O worker #140] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.387 [INFO - New I/O worker #140] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.396531 7437 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.380646 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40382 04:11:51.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.385037 (+ 4391us) server_negotiation.cc:184] Beginning negotiation 04:11:51.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.385060 (+ 23us) server_negotiation.cc:373] Waiting for connection header 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.385099 (+ 39us) server_negotiation.cc:381] Connection header received 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.385223 (+ 124us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.385246 (+ 23us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.385386 (+ 140us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.394536 (+ 9150us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.395789 (+ 1253us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40382: BlockingRecv error: recv got EOF from 127.0.0.1:40382 (error 108) 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":4098} 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.397222 7440 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.381342 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:57820 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.384208 (+ 2866us) server_negotiation.cc:184] Beginning negotiation 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.384228 (+ 20us) server_negotiation.cc:373] Waiting for connection header 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.384273 (+ 45us) server_negotiation.cc:381] Connection header received 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.384399 (+ 126us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.384422 (+ 23us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.384564 (+ 142us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.386707 (+ 2143us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.395222 (+ 8515us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57820: BlockingRecv error: recv got EOF from 127.0.0.1:57820 (error 108) 04:11:51.397 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":2567} 04:11:51.448 [DEBUG - New I/O worker #142] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.449 [INFO - New I/O worker #142] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.452 [DEBUG - New I/O worker #141] (Connection.java:691) [peer master-127.7.12.126:43389(127.7.12.126:43389)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.452 [INFO - New I/O worker #141] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.126:43389: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.451 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.451220 7437 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.443390 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40386 04:11:51.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.447875 (+ 4485us) server_negotiation.cc:184] Beginning negotiation 04:11:51.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.447891 (+ 16us) server_negotiation.cc:373] Waiting for connection header 04:11:51.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.447920 (+ 29us) server_negotiation.cc:381] Connection header received 04:11:51.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.448017 (+ 97us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.448031 (+ 14us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.448134 (+ 103us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.449574 (+ 1440us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.450054 (+ 480us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40386: BlockingRecv error: recv got EOF from 127.0.0.1:40386 (error 108) 04:11:51.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":4261} 04:11:51.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.454979 7440 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.443748 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:57824 04:11:51.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.446718 (+ 2970us) server_negotiation.cc:184] Beginning negotiation 04:11:51.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.446741 (+ 23us) server_negotiation.cc:373] Waiting for connection header 04:11:51.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.446783 (+ 42us) server_negotiation.cc:381] Connection header received 04:11:51.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.450635 (+ 3852us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.450660 (+ 25us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.450809 (+ 149us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.453066 (+ 2257us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.454247 (+ 1181us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57824: BlockingRecv error: recv got EOF from 127.0.0.1:57824 (error 108) 04:11:51.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":2679} 04:11:51.507 [DEBUG - New I/O worker #136] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.507 [INFO - New I/O worker #136] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.509125 7437 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.505664 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40390 04:11:51.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.506033 (+ 369us) server_negotiation.cc:184] Beginning negotiation 04:11:51.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.506057 (+ 24us) server_negotiation.cc:373] Waiting for connection header 04:11:51.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.506150 (+ 93us) server_negotiation.cc:381] Connection header received 04:11:51.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.506422 (+ 272us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.506448 (+ 26us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.506600 (+ 152us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.506760 (+ 160us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.508460 (+ 1700us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40390: BlockingRecv error: recv got EOF from 127.0.0.1:40390 (error 108) 04:11:51.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":87} 04:11:51.511 [DEBUG - New I/O worker #143] (Connection.java:691) [peer master-127.7.12.126:43389(127.7.12.126:43389)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.511 [INFO - New I/O worker #143] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.126:43389: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.513638 7440 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.504127 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:57828 04:11:51.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.510387 (+ 6260us) server_negotiation.cc:184] Beginning negotiation 04:11:51.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.510414 (+ 27us) server_negotiation.cc:373] Waiting for connection header 04:11:51.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.510455 (+ 41us) server_negotiation.cc:381] Connection header received 04:11:51.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.510578 (+ 123us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.510600 (+ 22us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.510740 (+ 140us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.512452 (+ 1712us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.512927 (+ 475us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57828: BlockingRecv error: recv got EOF from 127.0.0.1:57828 (error 108) 04:11:51.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":5954} 04:11:51.567 [DEBUG - New I/O worker #138] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.567 [INFO - New I/O worker #138] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.569577 7437 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.565943 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40394 04:11:51.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.566325 (+ 382us) server_negotiation.cc:184] Beginning negotiation 04:11:51.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.566345 (+ 20us) server_negotiation.cc:373] Waiting for connection header 04:11:51.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.566398 (+ 53us) server_negotiation.cc:381] Connection header received 04:11:51.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.566531 (+ 133us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.566552 (+ 21us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.566700 (+ 148us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.566851 (+ 151us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.568866 (+ 2015us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40394: BlockingRecv error: recv got EOF from 127.0.0.1:40394 (error 108) 04:11:51.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":90} 04:11:51.575 [DEBUG - New I/O worker #137] (Connection.java:691) [peer master-127.7.12.126:43389(127.7.12.126:43389)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.575 [INFO - New I/O worker #137] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.126:43389: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.577404 7440 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.565233 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:57832 04:11:51.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.572345 (+ 7112us) server_negotiation.cc:184] Beginning negotiation 04:11:51.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.572397 (+ 52us) server_negotiation.cc:373] Waiting for connection header 04:11:51.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.572464 (+ 67us) server_negotiation.cc:381] Connection header received 04:11:51.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.573857 (+ 1393us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.573892 (+ 35us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.574058 (+ 166us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.576229 (+ 2171us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.576683 (+ 454us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57832: BlockingRecv error: recv got EOF from 127.0.0.1:57832 (error 108) 04:11:51.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":5486} 04:11:51.630 [DEBUG - New I/O worker #139] (Connection.java:691) [peer master-127.7.12.126:43389(127.7.12.126:43389)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.630 [INFO - New I/O worker #139] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.126:43389: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.634 [DEBUG - New I/O worker #140] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.634 [INFO - New I/O worker #140] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.634456 7440 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.628787 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:57836 04:11:51.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.629231 (+ 444us) server_negotiation.cc:184] Beginning negotiation 04:11:51.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.629253 (+ 22us) server_negotiation.cc:373] Waiting for connection header 04:11:51.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.629314 (+ 61us) server_negotiation.cc:381] Connection header received 04:11:51.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.629451 (+ 137us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.629474 (+ 23us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.629620 (+ 146us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.629778 (+ 158us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.633754 (+ 3976us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57836: BlockingRecv error: recv got EOF from 127.0.0.1:57836 (error 108) 04:11:51.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":163} 04:11:51.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.636646 7437 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.631752 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40398 04:11:51.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.632288 (+ 536us) server_negotiation.cc:184] Beginning negotiation 04:11:51.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.632313 (+ 25us) server_negotiation.cc:373] Waiting for connection header 04:11:51.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.633540 (+ 1227us) server_negotiation.cc:381] Connection header received 04:11:51.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.633680 (+ 140us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.633703 (+ 23us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.633850 (+ 147us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.633993 (+ 143us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.635935 (+ 1942us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40398: BlockingRecv error: recv got EOF from 127.0.0.1:40398 (error 108) 04:11:51.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":271} 04:11:51.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:51.665833 7464 catalog_manager.cc:1120] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739: acquiring CA information for follower catalog manager: success 04:11:51.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:51.669096 7464 catalog_manager.cc:1148] T 00000000000000000000000000000000 P 1bfb2b1ddd09416391ffc0f9664ef739: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:51.697 [DEBUG - New I/O worker #142] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.698 [INFO - New I/O worker #142] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.699666 7437 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.687279 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40402 04:11:51.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.696841 (+ 9562us) server_negotiation.cc:184] Beginning negotiation 04:11:51.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.696866 (+ 25us) server_negotiation.cc:373] Waiting for connection header 04:11:51.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.696912 (+ 46us) server_negotiation.cc:381] Connection header received 04:11:51.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.697044 (+ 132us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.697067 (+ 23us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.697215 (+ 148us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.697372 (+ 157us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.698879 (+ 1507us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40402: BlockingRecv error: recv got EOF from 127.0.0.1:40402 (error 108) 04:11:51.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":9250} 04:11:51.710 [DEBUG - New I/O worker #141] (Negotiator.java:764) Authenticated connection [id: 0x2cb261f4, /127.0.0.1:57840 => /127.7.12.126:43389] using TOKEN/null 04:11:51.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:51.775311 7453 catalog_manager.cc:1120] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672: acquiring CA information for follower catalog manager: success 04:11:51.776 [DEBUG - New I/O worker #143] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.776 [INFO - New I/O worker #143] (ConnectToCluster.java:361) Unable to connect to master 127.7.12.124:41359: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because the server chose not to accept token authentication 04:11:51.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0627 04:11:51.778592 7437 negotiation.cc:313] Failed RPC negotiation. Trace: 04:11:51.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.768106 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40404 04:11:51.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.775392 (+ 7286us) server_negotiation.cc:184] Beginning negotiation 04:11:51.778 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.775425 (+ 33us) server_negotiation.cc:373] Waiting for connection header 04:11:51.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.775470 (+ 45us) server_negotiation.cc:381] Connection header received 04:11:51.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.775601 (+ 131us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 04:11:51.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.775622 (+ 21us) server_negotiation.cc:420] Received NEGOTIATE request from client 04:11:51.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.775760 (+ 138us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 04:11:51.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.777449 (+ 1689us) server_negotiation.cc:205] Negotiated authn=SASL 04:11:51.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0627 04:11:51.777886 (+ 437us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40404: BlockingRecv error: recv got EOF from 127.0.0.1:40404 (error 108) 04:11:51.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":6990} 04:11:51.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:51.778647 7453 catalog_manager.cc:1148] T 00000000000000000000000000000000 P dff5e6e71a684c1db979b7651f240672: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 04:11:51.860 [DEBUG - New I/O worker #136] (Negotiator.java:764) Authenticated connection [id: 0xb47f1566, /127.0.0.1:40406 => /127.7.12.124:41359] using TOKEN/null 04:11:51.868 [DEBUG - New I/O worker #138] (Connection.java:691) [peer master-127.7.12.125:41989(127.7.12.125:41989)] cleaning up while in state READY due to: connection disconnected 04:11:51.869 [DEBUG - New I/O worker #141] (Connection.java:691) [peer master-127.7.12.126:43389(127.7.12.126:43389)] cleaning up while in state READY due to: connection disconnected 04:11:51.871 [DEBUG - New I/O worker #136] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:11:51.875 [DEBUG - New I/O worker #128] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state READY due to: connection disconnected 04:11:51.882 [DEBUG - New I/O worker #136] (Connection.java:691) [peer master-127.7.12.124:41359(127.7.12.124:41359)] cleaning up while in state READY due to: connection disconnected 04:11:51.882 [DEBUG - New I/O worker #129] (Connection.java:691) [peer master-127.7.12.125:41989(127.7.12.125:41989)] cleaning up while in state READY due to: connection disconnected 04:11:51.883 [DEBUG - New I/O worker #127] (AsyncKuduClient.java:2487) Releasing all remaining resources 04:11:51.891 [DEBUG - New I/O worker #127] (Connection.java:691) [peer master-127.7.12.126:43389(127.7.12.126:43389)] cleaning up while in state READY due to: connection disconnected 04:11:51.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:51.891968 7217 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 7237 04:11:51.944 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:51.943816 7217 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 7304 04:11:51.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0627 04:11:51.961174 7217 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskusvpEt/build/tsan/bin/kudu-master with pid 7372 Time: 186.596 There was 1 failure: 1) testExternallyProvidedSubjectRefreshedExternally(org.apache.kudu.client.TestSecurity) org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=26, TimeoutTracker(timeout=30000, elapsed=29608), Traces: [0ms] refreshing cache from master, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [63ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [69ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581, [70ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [250ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: [peer master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception from downstream on [id: 0x2fae7299, /127.0.0.1:57014 => /127.0.202.125:43873], [282ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [336ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: [peer master-127.0.202.125:43873(127.0.202.125:43873)] unexpected exception from downstream on [id: 0x2fae7299, /127.0.0.1:57014 => /127.0.202.125:43873], [357ms] refreshing cache from master, [358ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [358ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [360ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [360ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [361ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [363ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [364ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [376ms] refreshing cache from master, [377ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [377ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [378ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [379ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581, [381ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [382ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [383ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [397ms] refreshing cache from master, [397ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [398ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [399ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581, [402ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [408ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [410ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [411ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [416ms] refreshing cache from master, [416ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [417ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [418ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [418ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [419ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [421ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [422ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [457ms] refreshing cache from master, [457ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [457ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [460ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [460ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [462ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [463ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [465ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [517ms] refreshing cache from master, [517ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [517ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [518ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [520ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [523ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [526ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [527ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [597ms] refreshing cache from master, [597ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [598ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [599ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [601ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581, [602ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [604ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [605ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.126:46581(127.0.202.126:46581): Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [736ms] refreshing cache from master, [737ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [737ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [738ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [739ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [741ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [743ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [744ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [797ms] refreshing cache from master, [797ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [798ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [798ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [801ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [802ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [802ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [804ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [1417ms] refreshing cache from master, [1417ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [1417ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [1421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [1421ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [1423ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [1423ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [1424ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873, [2537ms] refreshing cache from master, [2537ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [2537ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [2540ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [2541ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.126:46581: Network error: java.net.ConnectException: Connection refused: /127.0.202.126:46581, [2542ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.125:43873: Network error: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [2543ms] Sub RPC ConnectToMaster: received response from server master-127.0.202.124:43241: OK, [2543ms] delaying RPC due to: Service unavailable: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: Failed to connect to peer master-127.0.202.125:43873(127.0.202.125:43873): Connection refused: /127.0.202.125:43873, [3857ms] refreshing cache from master, [3857ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.126:46581, [3858ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.124:43241, [3861ms] Sub RPC ConnectToMaster: sending RPC to server master-127.0.202.125:43873, [3861ms] too many traces: truncated at 100 traces, deferred=Deferred@531938157(state=PENDING, result=null, callback=(continuation of Deferred@691636556 after retry RPC after error@610503197) -> (continuation of Deferred@1438562118 after retry RPC after error@1877676367) -> (continuation of Deferred@176002583 after retry RPC after error@855033383) -> (continuation of Deferred@315773880 after retry RPC after error@621038020) -> (continuation of Deferred@1127198301 after retry RPC after error@1206685665) -> (continuation of Deferred@1201883959 after retry RPC after error@1089368174) -> (continuation of Deferred@2114810728 after retry RPC after error@1564305615) -> (continuation of Deferred@1735340 after retry RPC after error@868489153) -> (continuation of Deferred@642229474 after retry RPC after error@269147125) -> (continuation of Deferred@1125522498 after retry RPC after error@1358918142) -> (continuation of Deferred@1258602423 after retry RPC after error@219615313) -> (continuation of Deferred@2115041725 after retry RPC after error@1430980352) -> (continuation of Deferred@1004335897 after retry RPC after error@243071403) -> (continuation of Deferred@2002269384 after retry RPC after error@2049334820) -> (continuation of Deferred@650001986 after retry RPC after error@1340484255) -> (continuation of Deferred@2033105375 after retry RPC after error@825298848) -> (continuation of Deferred@1722444778 after retry RPC after error@417732466) -> (continuation of Deferred@344634864 after retry RPC after error@763356366) -> (continuation of Deferred@406794160 after retry RPC after error@810214978) -> (continuation of Deferred@1907286158 after retry RPC after error@589636577) -> (continuation of Deferred@1107745434 after retry RPC after error@630803566) -> (continuation of Deferred@590705347 after retry RPC after error@696608623) -> (continuation of Deferred@1571007183 after retry RPC after error@850814761) -> (continuation of Deferred@983414998 after retry RPC after error@950083317) -> (continuation of Deferred@2014588668 after retry RPC after error@2040173048), errback=(continuation of Deferred@691636556 after retry RPC after error@610503197) -> (continuation of Deferred@1438562118 after retry RPC after error@1877676367) -> (continuation of Deferred@176002583 after retry RPC after error@855033383) -> (continuation of Deferred@315773880 after retry RPC after error@621038020) -> (continuation of Deferred@1127198301 after retry RPC after error@1206685665) -> (continuation of Deferred@1201883959 after retry RPC after error@1089368174) -> (continuation of Deferred@2114810728 after retry RPC after error@1564305615) -> (continuation of Deferred@1735340 after retry RPC after error@868489153) -> (continuation of Deferred@642229474 after retry RPC after error@269147125) -> (continuation of Deferred@1125522498 after retry RPC after error@1358918142) -> (continuation of Deferred@1258602423 after retry RPC after error@219615313) -> (continuation of Deferred@2115041725 after retry RPC after error@1430980352) -> (continuation of Deferred@1004335897 after retry RPC after error@243071403) -> (continuation of Deferred@2002269384 after retry RPC after error@2049334820) -> (continuation of Deferred@650001986 after retry RPC after error@1340484255) -> (continuation of Deferred@2033105375 after retry RPC after error@825298848) -> (continuation of Deferred@1722444778 after retry RPC after error@417732466) -> (continuation of Deferred@344634864 after retry RPC after error@763356366) -> (continuation of Deferred@406794160 after retry RPC after error@810214978) -> (continuation of Deferred@1907286158 after retry RPC after error@589636577) -> (continuation of Deferred@1107745434 after retry RPC after error@630803566) -> (continuation of Deferred@590705347 after retry RPC after error@696608623) -> (continuation of Deferred@1571007183 after retry RPC after error@850814761) -> (continuation of Deferred@983414998 after retry RPC after error@950083317) -> (continuation of Deferred@2014588668 after retry RPC after error@2040173048))) at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:413) at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:197) at org.apache.kudu.client.TestSecurity.checkClientCanReconnect(TestSecurity.java:135) at org.apache.kudu.client.TestSecurity.testExternallyProvidedSubjectRefreshedExternally(TestSecurity.java:431) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1676) at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2121) at org.apache.kudu.client.AsyncKuduClient.access$1200(AsyncKuduClient.java:261) at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1328) at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1318) at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:289) at org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49) at org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:323) at org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:312) at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275) at org.apache.kudu.client.KuduRpc.callback(KuduRpc.java:322) at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:296) at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59) at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149) at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145) at org.apache.kudu.client.Connection.messageReceived(Connection.java:390) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.0.202.126:46581,127.0.202.124:43241,127.0.202.125:43873) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.126:46581,org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873 at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:287) at org.apache.kudu.client.ConnectToCluster.access$100(ConnectToCluster.java:49) at org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:323) at org.apache.kudu.client.ConnectToCluster$ConnectToMasterCB.call(ConnectToCluster.java:312) at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:275) at org.apache.kudu.client.KuduRpc.callback(KuduRpc.java:322) at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:296) at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59) at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:149) at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:145) at org.apache.kudu.client.Connection.messageReceived(Connection.java:390) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more Caused by: org.apache.kudu.client.RecoverableException: java.net.ConnectException: Connection refused: /127.0.202.125:43873 at org.apache.kudu.client.Connection.channelClosed(Connection.java:266) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.timeout.ReadTimeoutHandler.channelClosed(ReadTimeoutHandler.java:176) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:60) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:493) at org.jboss.netty.handler.codec.frame.FrameDecoder.channelClosed(FrameDecoder.java:371) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:88) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireChannelClosed(Channels.java:468) at org.jboss.netty.channel.Channels$6.run(Channels.java:457) at org.jboss.netty.channel.socket.ChannelRunnableWrapper.run(ChannelRunnableWrapper.java:40) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315) ... 7 more FAILURES!!! Tests run: 9, Failures: 1