JUnit version 4.12 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpieVMq/java/build/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpieVMq/java/build/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/tmp/dist-test-taskpieVMq/java/build/jars/logback-classic-1.0.9.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 11:18:08.034 [INFO - main] (RandomUtils.java:46) Using random seed: 1554549488032 11:18:08.099 [INFO - main] (RetryRule.java:135) Creating RetryStatement with result reporter and retry count of 0 (no retries) .11:18:08.104 [INFO - main] (KuduTestHarness.java:139) Creating a new MiniKuduCluster... 11:18:08.112 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskpieVMq/test-tmp 11:18:08.128 [INFO - main] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/debug/bin 11:18:08.131 [INFO - main] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskpieVMq/java/build/dist-test/../../../build/debug/bin/kudu, test, mini_cluster, --serialization=pb] 11:18:08.327 [DEBUG - main] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 3 enable_kerberos: true cluster_root: "/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558" extra_master_flags: "--authn_token_validity_seconds=1" extra_master_flags: "--authz_token_validity_seconds=1" extra_master_flags: "--rpc_inject_invalid_authn_token_ratio=0.5" extra_tserver_flags: "--rpc_inject_invalid_authn_token_ratio=0.5" extra_tserver_flags: "--tserver_enforce_access_control=true" extra_tserver_flags: "--tserver_inject_invalid_authz_token_ratio=0.5" hms_mode: NONE mini_kdc_options { } } 11:18:08.338 [DEBUG - main] (MiniKuduCluster.java:165) Response: 11:18:08.339 [DEBUG - main] (MiniKuduCluster.java:155) Request: start_cluster { } 11:18:08.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Loading random data 11:18:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Initializing database '/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/principal' for realm 'KRBTEST.COM', 11:18:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) master key name 'K/M@KRBTEST.COM' 11:18:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:08 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): setting up network... 11:18:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:08 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) 11:18:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked 11:18:08.396 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 11:18:08.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:08 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): skipping unrecognized local address family 17 11:18:08.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:08 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): skipping unrecognized local address family 17 11:18:08.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:08 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): set up 1 sockets 11:18:08.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:08 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): commencing operation 11:18:08.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: starting... 11:18:08.734 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:08.733835 3134 mini_kdc.cc:119] Time spent starting KDC: real 0.385s user 0.000s sys 0.006s 11:18:08.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy 11:18:08.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 11:18:08.755 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-admin@KRBTEST.COM" created. 11:18:08.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy 11:18:08.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 11:18:08.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-user@KRBTEST.COM" created. 11:18:08.780 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy 11:18:08.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 11:18:08.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "joe-interloper@KRBTEST.COM" created. 11:18:08.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 11:18:08.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/test-user.keytab. 11:18:08.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/test-user.keytab. 11:18:08.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/test-user.keytab. 11:18:08.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/test-user.keytab. 11:18:08.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:08 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:08.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 11:18:08.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.3.15.190@KRBTEST.COM; defaulting to no policy 11:18:08.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:08.839 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.3.15.190@KRBTEST.COM" created. 11:18:08.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:08.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.190.keytab. 11:18:08.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.190.keytab. 11:18:08.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.190 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.190.keytab. 11:18:08.863 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.190 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.190.keytab. 11:18:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.863984 3134 external_mini_cluster.cc:933] Running /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master 11:18:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master 11:18:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/wal 11:18:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/data 11:18:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 11:18:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 11:18:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 11:18:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 11:18:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.15.190:34417 11:18:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.15.190 11:18:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:08.864 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 11:18:08.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:08.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:08.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:08.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:08.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 11:18:08.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:08.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/logs 11:18:08.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/data/info.pb 11:18:08.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:08.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 11:18:08.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 11:18:08.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 11:18:08.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.190.keytab 11:18:08.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.15.190 11:18:08.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:08.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:08.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:08.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=1 11:18:08.866 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authz_token_validity_seconds=1 11:18:08.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:08.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 11:18:08.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.3.15.190:34417,127.3.15.189:41941,127.3.15.188:39851 with env {KRB5CCNAME=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 11:18:08.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 11:18:08.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:08.916661 3153 flags.cc:404] Enabled unsafe flag: --rpc_inject_invalid_authn_token_ratio=0.5 11:18:08.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:08.916837 3153 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 11:18:08.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:08.916847 3153 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.3.15.190 11:18:08.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:08.916877 3153 flags.cc:404] Enabled unsafe flag: --never_fsync=true 11:18:08.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:08.917804 3153 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=1 11:18:08.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:08.917821 3153 flags.cc:404] Enabled experimental flag: --authz_token_validity_seconds=1 11:18:08.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:08.917827 3153 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 11:18:08.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:08.917840 3153 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 11:18:08.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:08.917845 3153 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 11:18:08.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:08.917853 3153 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 11:18:08.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:08.917863 3153 flags.cc:404] Enabled experimental flag: --principal=kudu/127.3.15.190 11:18:08.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.920678 3153 master_main.cc:73] Master server non-default flags: 11:18:08.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/data 11:18:08.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/wal 11:18:08.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=1 11:18:08.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authz_token_validity_seconds=1 11:18:08.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 11:18:08.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.3.15.190:34417,127.3.15.189:41941,127.3.15.188:39851 11:18:08.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:08.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:08.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 11:18:08.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.15.190:34417 11:18:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 11:18:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 11:18:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.190.keytab 11:18:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.15.190 11:18:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/data/info.pb 11:18:08.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.15.190 11:18:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 11:18:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --heap_profile_path=/tmp/kudu-master.3153 11:18:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 11:18:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 11:18:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/logs 11:18:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 11:18:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 11:18:08.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-cdh6.x-SNAPSHOT 11:18:08.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5327ca21851e41b4ad88c45c6cffc929a77ae8f4 11:18:08.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type DEBUG 11:18:08.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 06 Apr 2019 03:41:25 PST on quasar-mflytl-1.vpc.cloudera.com 11:18:08.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 304881 11:18:08.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.927598 3153 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 40854us 11:18:08.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:08 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549488, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:08.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.929260 3153 init.cc:359] Logged in from keytab as kudu/127.3.15.190@KRBTEST.COM (short username kudu) 11:18:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.930459 3153 fs_manager.cc:263] Metadata directory not provided 11:18:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.930510 3153 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 11:18:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.930745 3153 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 11:18:08.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.930773 3153 server_base.cc:434] Attempting to create new FS layout instead 11:18:08.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.933421 3153 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/data/instance: 11:18:08.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "63cbb0c1927d4d7dbc651859e214889d" 11:18:08.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:08 on dist-test-slave-dist-test-slave-6ztd" 11:18:08.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.933743 3153 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/wal/instance: 11:18:08.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "63cbb0c1927d4d7dbc651859e214889d" 11:18:08.933 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:08 on dist-test-slave-dist-test-slave-6ztd" 11:18:08.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.938719 3153 fs_manager.cc:503] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s 11:18:08.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.939309 3153 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 11:18:08.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.939682 3153 file_cache.cc:466] Constructed file cache lbm with capacity 419430 11:18:08.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.942466 3153 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 11:18:08.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.942540 3153 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/data,/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/wal 11:18:08.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "63cbb0c1927d4d7dbc651859e214889d" 11:18:08.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:08 on dist-test-slave-dist-test-slave-6ztd" 11:18:08.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.942617 3153 fs_report.cc:352] FS layout report 11:18:08.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 11:18:08.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/wal 11:18:08.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/wal 11:18:08.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/data/data 11:18:08.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 11:18:08.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 11:18:08.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 11:18:08.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 11:18:08.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 11:18:08.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 11:18:08.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 11:18:08.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 11:18:08.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 11:18:08.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 11:18:08.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.984532 3153 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 11:18:08.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.989151 3153 rpc_server.cc:205] RPC server started. Bound to: 127.3.15.190:34417 11:18:08.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.989840 3153 webserver.cc:308] Webserver started at http://127.3.15.190:33805/ using document root and password file 11:18:08.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.990344 3153 server_base.cc:608] Dumped server information to /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-0/data/info.pb 11:18:08.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.992897 3213 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:08.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.996625 3213 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } has no permanent_uuid. Determining permanent_uuid... 11:18:08.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:08.999079 3134 external_mini_cluster.cc:995] Started /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master as pid 3153 11:18:09.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:08 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549488, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.190@KRBTEST.COM for kudu/127.3.15.190@KRBTEST.COM 11:18:09.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.3.15.189@KRBTEST.COM; defaulting to no policy 11:18:09.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:09.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.3.15.189@KRBTEST.COM" created. 11:18:09.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.010279 3213 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } has no permanent_uuid. Determining permanent_uuid... 11:18:09.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.011270 3213 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.15.189:41941: Network error: Client connection negotiation failed: client connection to 127.3.15.189:41941: connect: Connection refused (error 111) 11:18:09.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.030843 3213 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } attempt: 1 11:18:09.031 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.031682 3213 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.15.189:41941: Network error: Client connection negotiation failed: client connection to 127.3.15.189:41941: connect: Connection refused (error 111) 11:18:09.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:09.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.189.keytab. 11:18:09.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.189.keytab. 11:18:09.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.189 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.189.keytab. 11:18:09.032 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.189 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.189.keytab. 11:18:09.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.033254 3134 external_mini_cluster.cc:933] Running /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master 11:18:09.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master 11:18:09.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/wal 11:18:09.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/data 11:18:09.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 11:18:09.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 11:18:09.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.15.189:41941 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.15.189 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/logs 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/data/info.pb 11:18:09.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.189.keytab 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.15.189 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=1 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authz_token_validity_seconds=1 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 11:18:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.3.15.190:34417,127.3.15.189:41941,127.3.15.188:39851 with env {KRB5CCNAME=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 11:18:09.080 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.079864 3213 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } attempt: 2 11:18:09.081 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.080935 3213 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.15.189:41941: Network error: Client connection negotiation failed: client connection to 127.3.15.189:41941: connect: Connection refused (error 111) 11:18:09.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 11:18:09.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.088625 3218 flags.cc:404] Enabled unsafe flag: --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.088804 3218 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 11:18:09.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.088815 3218 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.3.15.189 11:18:09.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.088843 3218 flags.cc:404] Enabled unsafe flag: --never_fsync=true 11:18:09.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.089974 3218 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=1 11:18:09.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.089992 3218 flags.cc:404] Enabled experimental flag: --authz_token_validity_seconds=1 11:18:09.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.089998 3218 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 11:18:09.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.090009 3218 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 11:18:09.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.090040 3218 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 11:18:09.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.090049 3218 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 11:18:09.090 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.090059 3218 flags.cc:404] Enabled experimental flag: --principal=kudu/127.3.15.189 11:18:09.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.092274 3218 master_main.cc:73] Master server non-default flags: 11:18:09.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/data 11:18:09.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/wal 11:18:09.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=1 11:18:09.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authz_token_validity_seconds=1 11:18:09.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.3.15.190:34417,127.3.15.189:41941,127.3.15.188:39851 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.15.189:41941 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.189.keytab 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.15.189 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:09.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/data/info.pb 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.15.189 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --heap_profile_path=/tmp/kudu-master.3218 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/logs 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-cdh6.x-SNAPSHOT 11:18:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5327ca21851e41b4ad88c45c6cffc929a77ae8f4 11:18:09.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type DEBUG 11:18:09.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 06 Apr 2019 03:41:25 PST on quasar-mflytl-1.vpc.cloudera.com 11:18:09.095 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 304881 11:18:09.099 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.098965 3218 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 41354us 11:18:09.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:09.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.100697 3218 init.cc:359] Logged in from keytab as kudu/127.3.15.189@KRBTEST.COM (short username kudu) 11:18:09.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.101770 3218 fs_manager.cc:263] Metadata directory not provided 11:18:09.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.101812 3218 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 11:18:09.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.102054 3218 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 11:18:09.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.102072 3218 server_base.cc:434] Attempting to create new FS layout instead 11:18:09.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.103813 3218 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/data/instance: 11:18:09.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" 11:18:09.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.104086 3218 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/wal/instance: 11:18:09.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" 11:18:09.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.109215 3218 fs_manager.cc:503] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s 11:18:09.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.109725 3218 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 11:18:09.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.110126 3218 file_cache.cc:466] Constructed file cache lbm with capacity 419430 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.112953 3218 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.113054 3218 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/data,/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/wal 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.113127 3218 fs_report.cc:352] FS layout report 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/wal 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/wal 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/data/data 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 11:18:09.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 11:18:09.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 11:18:09.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 11:18:09.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 11:18:09.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 11:18:09.156 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.155864 3218 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 11:18:09.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.160480 3218 rpc_server.cc:205] RPC server started. Bound to: 127.3.15.189:41941 11:18:09.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.161258 3218 webserver.cc:308] Webserver started at http://127.3.15.189:40531/ using document root and password file 11:18:09.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.161772 3218 server_base.cc:608] Dumped server information to /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-1/data/info.pb 11:18:09.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.164203 3278 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:09.167 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.167397 3278 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } has no permanent_uuid. Determining permanent_uuid... 11:18:09.169 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.169783 3134 external_mini_cluster.cc:995] Started /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master as pid 3218 11:18:09.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.189@KRBTEST.COM for kudu/127.3.15.190@KRBTEST.COM 11:18:09.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.3.15.188@KRBTEST.COM; defaulting to no policy 11:18:09.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.178002 3278 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } has no permanent_uuid. Determining permanent_uuid... 11:18:09.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:09.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.3.15.188@KRBTEST.COM" created. 11:18:09.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.189@KRBTEST.COM for kudu/127.3.15.189@KRBTEST.COM 11:18:09.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.185920 3278 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } has no permanent_uuid. Determining permanent_uuid... 11:18:09.186 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.186563 3278 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.15.188:39851: Network error: Client connection negotiation failed: client connection to 127.3.15.188:39851: connect: Connection refused (error 111) 11:18:09.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.192116 3213 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } attempt: 3 11:18:09.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549488, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.190@KRBTEST.COM for kudu/127.3.15.189@KRBTEST.COM 11:18:09.198 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.198830 3213 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } has no permanent_uuid. Determining permanent_uuid... 11:18:09.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.199528 3213 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.15.188:39851: Network error: Client connection negotiation failed: client connection to 127.3.15.188:39851: connect: Connection refused (error 111) 11:18:09.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:09.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.188.keytab. 11:18:09.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.188.keytab. 11:18:09.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.188 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.188.keytab. 11:18:09.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.188 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.188.keytab. 11:18:09.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.203724 3134 external_mini_cluster.cc:933] Running /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master 11:18:09.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/wal 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/data 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.15.188:39851 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.15.188 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:09.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/logs 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/data/info.pb 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.188.keytab 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.15.188 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=1 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authz_token_validity_seconds=1 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 11:18:09.205 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.3.15.190:34417,127.3.15.189:41941,127.3.15.188:39851 with env {KRB5CCNAME=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 11:18:09.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.205757 3278 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } attempt: 1 11:18:09.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.206282 3278 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.15.188:39851: Network error: Client connection negotiation failed: client connection to 127.3.15.188:39851: connect: Connection refused (error 111) 11:18:09.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.254458 3278 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } attempt: 2 11:18:09.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.255318 3278 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.15.188:39851: Network error: Client connection negotiation failed: client connection to 127.3.15.188:39851: connect: Connection refused (error 111) 11:18:09.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.263733 3213 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } attempt: 1 11:18:09.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.264502 3213 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.15.188:39851: Network error: Client connection negotiation failed: client connection to 127.3.15.188:39851: connect: Connection refused (error 111) 11:18:09.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 11:18:09.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.266336 3283 flags.cc:404] Enabled unsafe flag: --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.266562 3283 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 11:18:09.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.266578 3283 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.3.15.188 11:18:09.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.266623 3283 flags.cc:404] Enabled unsafe flag: --never_fsync=true 11:18:09.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.267941 3283 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=1 11:18:09.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.267971 3283 flags.cc:404] Enabled experimental flag: --authz_token_validity_seconds=1 11:18:09.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.267980 3283 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 11:18:09.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.267994 3283 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 11:18:09.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.268000 3283 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 11:18:09.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.268009 3283 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 11:18:09.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.268049 3283 flags.cc:404] Enabled experimental flag: --principal=kudu/127.3.15.188 11:18:09.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.270431 3283 master_main.cc:73] Master server non-default flags: 11:18:09.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/data 11:18:09.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/wal 11:18:09.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=1 11:18:09.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authz_token_validity_seconds=1 11:18:09.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 11:18:09.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.3.15.190:34417,127.3.15.189:41941,127.3.15.188:39851 11:18:09.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.15.188:39851 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.188.keytab 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.15.188 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/data/info.pb 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.15.188 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --heap_profile_path=/tmp/kudu-master.3283 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/logs 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-cdh6.x-SNAPSHOT 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5327ca21851e41b4ad88c45c6cffc929a77ae8f4 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type DEBUG 11:18:09.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 06 Apr 2019 03:41:25 PST on quasar-mflytl-1.vpc.cloudera.com 11:18:09.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 304881 11:18:09.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.276825 3283 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 41354us 11:18:09.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:09.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.278226 3283 init.cc:359] Logged in from keytab as kudu/127.3.15.188@KRBTEST.COM (short username kudu) 11:18:09.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.279103 3283 fs_manager.cc:263] Metadata directory not provided 11:18:09.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.279146 3283 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 11:18:09.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.279326 3283 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 11:18:09.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.279340 3283 server_base.cc:434] Attempting to create new FS layout instead 11:18:09.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.281235 3283 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/data/instance: 11:18:09.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "728d144d58ef4a2a8ec4f57856ae473b" 11:18:09.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.281482 3283 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/wal/instance: 11:18:09.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "728d144d58ef4a2a8ec4f57856ae473b" 11:18:09.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.286223 3283 fs_manager.cc:503] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s 11:18:09.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.286870 3283 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 11:18:09.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.287338 3283 file_cache.cc:466] Constructed file cache lbm with capacity 419430 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.290141 3283 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.290223 3283 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/data,/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/wal 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "728d144d58ef4a2a8ec4f57856ae473b" 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.290313 3283 fs_report.cc:352] FS layout report 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/wal 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/wal 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/data/data 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 11:18:09.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 11:18:09.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 11:18:09.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 11:18:09.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.320540 3283 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 11:18:09.324 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.324458 3283 rpc_server.cc:205] RPC server started. Bound to: 127.3.15.188:39851 11:18:09.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.325285 3283 webserver.cc:308] Webserver started at http://127.3.15.188:42955/ using document root and password file 11:18:09.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.325831 3283 server_base.cc:608] Dumped server information to /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/master-2/data/info.pb 11:18:09.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.328218 3343 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:09.330 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.330335 3134 external_mini_cluster.cc:995] Started /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master as pid 3283 11:18:09.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.331396 3343 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } has no permanent_uuid. Determining permanent_uuid... 11:18:09.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.3.15.129@KRBTEST.COM; defaulting to no policy 11:18:09.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:09.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.3.15.129@KRBTEST.COM" created. 11:18:09.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.188@KRBTEST.COM for kudu/127.3.15.190@KRBTEST.COM 11:18:09.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.343700 3213 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } attempt: 2 11:18:09.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.346174 3343 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } has no permanent_uuid. Determining permanent_uuid... 11:18:09.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.188@KRBTEST.COM for kudu/127.3.15.189@KRBTEST.COM 11:18:09.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549488, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.190@KRBTEST.COM for kudu/127.3.15.188@KRBTEST.COM 11:18:09.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.355724 3343 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } has no permanent_uuid. Determining permanent_uuid... 11:18:09.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.359059 3213 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d: Bootstrap starting. 11:18:09.359 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.188@KRBTEST.COM for kudu/127.3.15.188@KRBTEST.COM 11:18:09.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.361860 3213 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d: No blocks or log segments found. Creating new log. 11:18:09.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.362783 3213 log.cc:525] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d: Log is configured to *not* fsync() on all Append() calls 11:18:09.364 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.364692 3213 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d: No bootstrap required, opened a new log 11:18:09.365 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.365573 3343 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b: Bootstrap starting. 11:18:09.366 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.366515 3278 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } attempt: 3 11:18:09.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.367882 3343 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b: No blocks or log segments found. Creating new log. 11:18:09.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.367929 3213 raft_consensus.cc:340] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } 11:18:09.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.367986 3213 raft_consensus.cc:366] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:09.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.368064 3213 raft_consensus.cc:705] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cbb0c1927d4d7dbc651859e214889d, State: Initialized, Role: FOLLOWER 11:18:09.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.368552 3213 consensus_queue.cc:229] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } 11:18:09.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:09.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.129.keytab. 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.129.keytab. 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.129 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.129.keytab. 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.129 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.129.keytab. 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.368911 3213 sys_catalog.cc:337] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.368933 3213 sys_catalog.cc:340] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [sys.catalog]: This master's current role is: FOLLOWER 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.369175 3349 sys_catalog.cc:337] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.369199 3349 sys_catalog.cc:340] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [sys.catalog]: This master's current role is: FOLLOWER 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.369227 3343 log.cc:525] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b: Log is configured to *not* fsync() on all Append() calls 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.369575 3134 external_mini_cluster.cc:933] Running /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/data 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.15.129:0 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.3.15.129 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.15.129 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.3.15.190:34417,127.3.15.189:41941,127.3.15.188:39851 11:18:09.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/logs 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/data/info.pb 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.129.keytab 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.15.129 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:09.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_enforce_access_control=true 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_inject_invalid_authz_token_ratio=0.5 with env {KRB5CCNAME=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.370160 3343 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b: No bootstrap required, opened a new log 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.189@KRBTEST.COM for kudu/127.3.15.188@KRBTEST.COM 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.372125 3213 sys_catalog.cc:424] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [sys.catalog]: configured and running, proceeding with master startup. 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.372735 3343 raft_consensus.cc:340] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.372812 3343 raft_consensus.cc:366] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.372853 3343 raft_consensus.cc:705] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 728d144d58ef4a2a8ec4f57856ae473b, State: Initialized, Role: FOLLOWER 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.373301 3343 consensus_queue.cc:229] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.373556 3343 sys_catalog.cc:337] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.373581 3343 sys_catalog.cc:340] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [sys.catalog]: This master's current role is: FOLLOWER 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.374413 3343 sys_catalog.cc:424] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [sys.catalog]: configured and running, proceeding with master startup. 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.374765 3353 sys_catalog.cc:337] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.374784 3353 sys_catalog.cc:340] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [sys.catalog]: This master's current role is: FOLLOWER 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.377449 3278 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: Bootstrap starting. 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.379395 3278 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: No blocks or log segments found. Creating new log. 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.380153 3278 log.cc:525] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: Log is configured to *not* fsync() on all Append() calls 11:18:09.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.381424 3354 catalog_manager.cc:1113] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b: acquiring CA information for follower catalog manager: Not found: root CA entry not found 11:18:09.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.381449 3354 catalog_manager.cc:594] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 11:18:09.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.394976 3352 catalog_manager.cc:1113] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d: acquiring CA information for follower catalog manager: Not found: root CA entry not found 11:18:09.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.395046 3352 catalog_manager.cc:594] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 11:18:09.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.409593 3278 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: No bootstrap required, opened a new log 11:18:09.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.412297 3278 raft_consensus.cc:340] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } 11:18:09.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.412364 3278 raft_consensus.cc:366] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:09.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.412398 3278 raft_consensus.cc:705] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04f897cb6ac4fe38cfe92cc0220fa7c, State: Initialized, Role: FOLLOWER 11:18:09.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.412895 3278 consensus_queue.cc:229] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } 11:18:09.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.413261 3278 sys_catalog.cc:337] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.413292 3278 sys_catalog.cc:340] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [sys.catalog]: This master's current role is: FOLLOWER 11:18:09.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.414139 3278 sys_catalog.cc:424] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [sys.catalog]: configured and running, proceeding with master startup. 11:18:09.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.420785 3358 sys_catalog.cc:337] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.420811 3358 sys_catalog.cc:340] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [sys.catalog]: This master's current role is: FOLLOWER 11:18:09.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.423542 3359 catalog_manager.cc:1113] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: acquiring CA information for follower catalog manager: Not found: root CA entry not found 11:18:09.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.423585 3359 catalog_manager.cc:594] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 11:18:09.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.486096 3169 tablet.cc:1786] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d: Can't schedule compaction. Clean time has not been advanced past its initial value. 11:18:09.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.503976 3358 raft_consensus.cc:472] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 11:18:09.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.504207 3358 raft_consensus.cc:494] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } 11:18:09.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.505336 3358 leader_election.cc:296] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63cbb0c1927d4d7dbc651859e214889d (127.3.15.190:34417), 728d144d58ef4a2a8ec4f57856ae473b (127.3.15.188:39851) 11:18:09.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.506180 3320 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "728d144d58ef4a2a8ec4f57856ae473b" is_pre_election: true 11:18:09.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.506182 3190 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63cbb0c1927d4d7dbc651859e214889d" is_pre_election: true 11:18:09.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.506404 3320 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04f897cb6ac4fe38cfe92cc0220fa7c in term 0. 11:18:09.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.506526 3190 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a04f897cb6ac4fe38cfe92cc0220fa7c in term 0. 11:18:09.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.506927 3226 leader_election.cc:310] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 728d144d58ef4a2a8ec4f57856ae473b, a04f897cb6ac4fe38cfe92cc0220fa7c; no voters: 11:18:09.507 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.507297 3358 raft_consensus.cc:2640] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [term 0 FOLLOWER]: Leader pre-election won for term 1 11:18:09.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.507333 3358 raft_consensus.cc:472] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 11:18:09.508 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.507347 3358 raft_consensus.cc:2886] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [term 0 FOLLOWER]: Advancing to term 1 11:18:09.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.508534 3358 raft_consensus.cc:494] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } 11:18:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.508802 3358 leader_election.cc:296] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [CANDIDATE]: Term 1 election: Requested vote from peers 63cbb0c1927d4d7dbc651859e214889d (127.3.15.190:34417), 728d144d58ef4a2a8ec4f57856ae473b (127.3.15.188:39851) 11:18:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.509553 3190 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63cbb0c1927d4d7dbc651859e214889d" 11:18:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.509605 3190 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [term 0 FOLLOWER]: Advancing to term 1 11:18:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.510665 3190 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04f897cb6ac4fe38cfe92cc0220fa7c in term 1. 11:18:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.510962 3228 leader_election.cc:310] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63cbb0c1927d4d7dbc651859e214889d, a04f897cb6ac4fe38cfe92cc0220fa7c; no voters: 11:18:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.511234 3320 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "728d144d58ef4a2a8ec4f57856ae473b" 11:18:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.511314 3320 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [term 0 FOLLOWER]: Advancing to term 1 11:18:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.512414 3320 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a04f897cb6ac4fe38cfe92cc0220fa7c in term 1. 11:18:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.512760 3358 raft_consensus.cc:2640] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [term 1 FOLLOWER]: Leader election won for term 1 11:18:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.512976 3358 raft_consensus.cc:667] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [term 1 LEADER]: Becoming Leader. State: Replica: a04f897cb6ac4fe38cfe92cc0220fa7c, State: Running, Role: LEADER 11:18:09.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.513490 3358 consensus_queue.cc:206] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } 11:18:09.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.516424 3358 sys_catalog.cc:337] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a04f897cb6ac4fe38cfe92cc0220fa7c. Latest consensus state: current_term: 1 leader_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.516448 3358 sys_catalog.cc:340] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [sys.catalog]: This master's current role is: LEADER 11:18:09.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.516821 3363 catalog_manager.cc:1055] Loading table and tablet metadata into memory... 11:18:09.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.517951 3363 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... 11:18:09.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 11:18:09.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.518692 3351 flags.cc:404] Enabled unsafe flag: --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.518867 3351 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 11:18:09.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.518879 3351 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.3.15.129 11:18:09.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.518927 3351 flags.cc:404] Enabled unsafe flag: --never_fsync=true 11:18:09.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.520077 3351 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 11:18:09.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.520103 3351 flags.cc:404] Enabled experimental flag: --principal=kudu/127.3.15.129 11:18:09.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.520141 3351 flags.cc:404] Enabled experimental flag: --tserver_enforce_access_control=true 11:18:09.521 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.520174 3351 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.15.129 11:18:09.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.526507 3351 tablet_server_main.cc:78] Tablet server non-default flags: 11:18:09.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/data 11:18:09.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal 11:18:09.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:09.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.15.129:0 11:18:09.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 11:18:09.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.129.keytab 11:18:09.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.15.129 11:18:09.526 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/data/info.pb 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.15.129 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.3.15.190:34417,127.3.15.189:41941,127.3.15.188:39851 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_enforce_access_control=true 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_inject_invalid_authz_token_ratio=0.5 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --heap_profile_path=/tmp/kudu-tserver.3351 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.3.15.129 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/logs 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-cdh6.x-SNAPSHOT 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5327ca21851e41b4ad88c45c6cffc929a77ae8f4 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type DEBUG 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 06 Apr 2019 03:41:25 PST on quasar-mflytl-1.vpc.cloudera.com 11:18:09.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 304881 11:18:09.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.544420 3351 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 41354us 11:18:09.549 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:09.550 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.550140 3351 init.cc:359] Logged in from keytab as kudu/127.3.15.129@KRBTEST.COM (short username kudu) 11:18:09.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.551111 3351 fs_manager.cc:263] Metadata directory not provided 11:18:09.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.551298 3351 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 11:18:09.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.551617 3351 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 11:18:09.551 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.551643 3351 server_base.cc:434] Attempting to create new FS layout instead 11:18:09.553 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.553863 3351 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/data/instance: 11:18:09.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "3ac72b4ba12248ddad09193985a1e867" 11:18:09.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.554196 3351 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal/instance: 11:18:09.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "3ac72b4ba12248ddad09193985a1e867" 11:18:09.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.560174 3351 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s 11:18:09.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.560910 3351 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 11:18:09.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.561472 3351 file_cache.cc:466] Constructed file cache lbm with capacity 419430 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.564239 3351 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.564347 3351 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/data,/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "3ac72b4ba12248ddad09193985a1e867" 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.564442 3351 fs_report.cc:352] FS layout report 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/data/data 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 11:18:09.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 11:18:09.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 11:18:09.565 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 11:18:09.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.571691 3299 tablet.cc:1786] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b: Can't schedule compaction. Clean time has not been advanced past its initial value. 11:18:09.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.593616 3351 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 11:18:09.594 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.594847 3351 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) 11:18:09.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.594874 3351 ts_tablet_manager.cc:376] Registered 0 tablets 11:18:09.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.601475 3351 rpc_server.cc:205] RPC server started. Bound to: 127.3.15.129:33105 11:18:09.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.602267 3351 webserver.cc:308] Webserver started at http://127.3.15.129:46453/ using document root and password file 11:18:09.602 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.602828 3351 server_base.cc:608] Dumped server information to /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/data/info.pb 11:18:09.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.607678 3134 external_mini_cluster.cc:995] Started /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver as pid 3351 11:18:09.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.608721 3320 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [term 1 FOLLOWER]: Refusing update from remote peer a04f897cb6ac4fe38cfe92cc0220fa7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 11:18:09.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.608922 3190 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [term 1 FOLLOWER]: Refusing update from remote peer a04f897cb6ac4fe38cfe92cc0220fa7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 11:18:09.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.609791 3361 consensus_queue.cc:984] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:09.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.610177 3358 consensus_queue.cc:984] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:09.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.614708 3349 sys_catalog.cc:337] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a04f897cb6ac4fe38cfe92cc0220fa7c. Latest consensus state: current_term: 1 leader_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.614747 3349 sys_catalog.cc:340] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [sys.catalog]: This master's current role is: FOLLOWER 11:18:09.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.615308 3353 sys_catalog.cc:337] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a04f897cb6ac4fe38cfe92cc0220fa7c. Latest consensus state: current_term: 1 leader_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.615339 3353 sys_catalog.cc:340] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [sys.catalog]: This master's current role is: FOLLOWER 11:18:09.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.616516 3361 sys_catalog.cc:337] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.616544 3361 sys_catalog.cc:340] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [sys.catalog]: This master's current role is: LEADER 11:18:09.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.617059 3349 sys_catalog.cc:337] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.617085 3349 sys_catalog.cc:340] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d [sys.catalog]: This master's current role is: FOLLOWER 11:18:09.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.619127 3361 sys_catalog.cc:337] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.619153 3361 sys_catalog.cc:340] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c [sys.catalog]: This master's current role is: LEADER 11:18:09.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.619670 3363 catalog_manager.cc:950] Generated new certificate authority record 11:18:09.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.620318 3353 sys_catalog.cc:337] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } 11:18:09.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.620347 3353 sys_catalog.cc:340] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b [sys.catalog]: This master's current role is: FOLLOWER 11:18:09.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.620884 3490 mvcc.cc:203] Tried to move safe_time back from 6367434709428748288 to 6367434709053894656. Current Snapshot: MvccSnapshot[committed={T|T < 6367434709428748288}] 11:18:09.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.620934 3363 catalog_manager.cc:1075] Loading token signing keys... 11:18:09.626 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.3.15.130@KRBTEST.COM; defaulting to no policy 11:18:09.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.629163 3363 catalog_manager.cc:4280] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: Generated new TSK 0 11:18:09.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:09.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.3.15.130@KRBTEST.COM" created. 11:18:09.635 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.129@KRBTEST.COM for kudu/127.3.15.189@KRBTEST.COM 11:18:09.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.129@KRBTEST.COM for kudu/127.3.15.190@KRBTEST.COM 11:18:09.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.129@KRBTEST.COM for kudu/127.3.15.188@KRBTEST.COM 11:18:09.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.641988 3484 heartbeater.cc:345] Connected to a master server at 127.3.15.189:41941 11:18:09.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.642180 3484 heartbeater.cc:426] Registering TS with master... 11:18:09.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.642525 3484 heartbeater.cc:475] Master 127.3.15.189:41941 requested a full tablet report, sending... 11:18:09.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.643185 3485 heartbeater.cc:345] Connected to a master server at 127.3.15.190:34417 11:18:09.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.643234 3485 heartbeater.cc:426] Registering TS with master... 11:18:09.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.643347 3485 heartbeater.cc:475] Master 127.3.15.190:34417 requested a full tablet report, sending... 11:18:09.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.643811 3245 ts_manager.cc:159] Registered new tserver with Master: 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) 11:18:09.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.644418 3177 ts_manager.cc:159] Registered new tserver with Master: 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) 11:18:09.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.645571 3245 master_service.cc:271] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.15.129@KRBTEST.COM'} at 127.3.15.129:40081 11:18:09.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.645897 3483 heartbeater.cc:345] Connected to a master server at 127.3.15.188:39851 11:18:09.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.645943 3483 heartbeater.cc:426] Registering TS with master... 11:18:09.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.646090 3483 heartbeater.cc:475] Master 127.3.15.188:39851 requested a full tablet report, sending... 11:18:09.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.646948 3310 ts_manager.cc:159] Registered new tserver with Master: 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) 11:18:09.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:09.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.130.keytab. 11:18:09.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.130.keytab. 11:18:09.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.130 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.130.keytab. 11:18:09.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.130 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.130.keytab. 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.663337 3134 external_mini_cluster.cc:933] Running /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/data 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.15.130:0 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.3.15.130 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.15.130 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.3.15.190:34417,127.3.15.189:41941,127.3.15.188:39851 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/logs 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/data/info.pb 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 11:18:09.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 11:18:09.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 11:18:09.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.130.keytab 11:18:09.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.15.130 11:18:09.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:09.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:09.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:09.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_enforce_access_control=true 11:18:09.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_inject_invalid_authz_token_ratio=0.5 with env {KRB5CCNAME=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 11:18:09.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 11:18:09.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.708077 3501 flags.cc:404] Enabled unsafe flag: --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.708245 3501 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 11:18:09.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.708256 3501 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.3.15.130 11:18:09.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.708289 3501 flags.cc:404] Enabled unsafe flag: --never_fsync=true 11:18:09.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.709148 3501 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 11:18:09.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.709183 3501 flags.cc:404] Enabled experimental flag: --principal=kudu/127.3.15.130 11:18:09.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.709224 3501 flags.cc:404] Enabled experimental flag: --tserver_enforce_access_control=true 11:18:09.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.709256 3501 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.15.130 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.711318 3501 tablet_server_main.cc:78] Tablet server non-default flags: 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/data 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.15.130:0 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.130.keytab 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.15.130 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:09.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/data/info.pb 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.15.130 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.3.15.190:34417,127.3.15.189:41941,127.3.15.188:39851 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_enforce_access_control=true 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_inject_invalid_authz_token_ratio=0.5 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --heap_profile_path=/tmp/kudu-tserver.3501 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.3.15.130 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/logs 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-cdh6.x-SNAPSHOT 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5327ca21851e41b4ad88c45c6cffc929a77ae8f4 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type DEBUG 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 06 Apr 2019 03:41:25 PST on quasar-mflytl-1.vpc.cloudera.com 11:18:09.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 304881 11:18:09.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.718083 3501 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 41354us 11:18:09.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:09.719 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.719509 3501 init.cc:359] Logged in from keytab as kudu/127.3.15.130@KRBTEST.COM (short username kudu) 11:18:09.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.720378 3501 fs_manager.cc:263] Metadata directory not provided 11:18:09.720 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.720427 3501 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 11:18:09.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.721092 3501 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 11:18:09.721 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.721125 3501 server_base.cc:434] Attempting to create new FS layout instead 11:18:09.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.722812 3501 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/data/instance: 11:18:09.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "73918a1b139243ff837530f26d7808c9" 11:18:09.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.723161 3501 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal/instance: 11:18:09.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "73918a1b139243ff837530f26d7808c9" 11:18:09.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.727588 3501 fs_manager.cc:503] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s 11:18:09.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.728159 3501 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 11:18:09.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.728516 3501 file_cache.cc:466] Constructed file cache lbm with capacity 419430 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.731040 3501 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.731127 3501 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/data,/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "73918a1b139243ff837530f26d7808c9" 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.731190 3501 fs_report.cc:352] FS layout report 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/data/data 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 11:18:09.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 11:18:09.803 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.803433 3501 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 11:18:09.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.804473 3501 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) 11:18:09.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.804500 3501 ts_tablet_manager.cc:376] Registered 0 tablets 11:18:09.810 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.810495 3501 rpc_server.cc:205] RPC server started. Bound to: 127.3.15.130:37203 11:18:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.811161 3501 webserver.cc:308] Webserver started at http://127.3.15.130:46425/ using document root and password file 11:18:09.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.811560 3501 server_base.cc:608] Dumped server information to /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/data/info.pb 11:18:09.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.817827 3134 external_mini_cluster.cc:995] Started /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver as pid 3501 11:18:09.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.130@KRBTEST.COM for kudu/127.3.15.188@KRBTEST.COM 11:18:09.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.130@KRBTEST.COM for kudu/127.3.15.190@KRBTEST.COM 11:18:09.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.130@KRBTEST.COM for kudu/127.3.15.189@KRBTEST.COM 11:18:09.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.3.15.131@KRBTEST.COM; defaulting to no policy 11:18:09.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.826491 3623 heartbeater.cc:345] Connected to a master server at 127.3.15.190:34417 11:18:09.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.826606 3623 heartbeater.cc:426] Registering TS with master... 11:18:09.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.826808 3621 heartbeater.cc:345] Connected to a master server at 127.3.15.188:39851 11:18:09.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.826848 3621 heartbeater.cc:426] Registering TS with master... 11:18:09.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.826891 3623 heartbeater.cc:475] Master 127.3.15.190:34417 requested a full tablet report, sending... 11:18:09.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.826959 3621 heartbeater.cc:475] Master 127.3.15.188:39851 requested a full tablet report, sending... 11:18:09.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.826891 3622 heartbeater.cc:345] Connected to a master server at 127.3.15.189:41941 11:18:09.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.827004 3622 heartbeater.cc:426] Registering TS with master... 11:18:09.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.827126 3622 heartbeater.cc:475] Master 127.3.15.189:41941 requested a full tablet report, sending... 11:18:09.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.827852 3245 ts_manager.cc:159] Registered new tserver with Master: 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) 11:18:09.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.827910 3310 ts_manager.cc:159] Registered new tserver with Master: 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) 11:18:09.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.827910 3177 ts_manager.cc:159] Registered new tserver with Master: 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) 11:18:09.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:09.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.3.15.131@KRBTEST.COM" created. 11:18:09.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.828861 3245 master_service.cc:271] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.15.130@KRBTEST.COM'} at 127.3.15.130:55465 11:18:09.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.131.keytab. 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.131.keytab. 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.131 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.131.keytab. 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.15.131 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.131.keytab. 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.852536 3134 external_mini_cluster.cc:933] Running /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/data 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.15.131:0 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.3.15.131 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.15.131 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.3.15.190:34417,127.3.15.189:41941,127.3.15.188:39851 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:09.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/logs 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/data/info.pb 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.131.keytab 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.15.131 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_enforce_access_control=true 11:18:09.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_inject_invalid_authz_token_ratio=0.5 with env {KRB5CCNAME=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 11:18:09.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 11:18:09.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.897119 3630 flags.cc:404] Enabled unsafe flag: --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.897294 3630 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 11:18:09.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.897307 3630 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.3.15.131 11:18:09.897 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.897353 3630 flags.cc:404] Enabled unsafe flag: --never_fsync=true 11:18:09.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.898306 3630 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 11:18:09.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.898344 3630 flags.cc:404] Enabled experimental flag: --principal=kudu/127.3.15.131 11:18:09.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.898385 3630 flags.cc:404] Enabled experimental flag: --tserver_enforce_access_control=true 11:18:09.898 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:09.898417 3630 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.15.131 11:18:09.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.900454 3630 tablet_server_main.cc:78] Tablet server non-default flags: 11:18:09.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/data 11:18:09.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal 11:18:09.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:09.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:09.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.15.131:0 11:18:09.900 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kudu_127.3.15.131.keytab 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.15.131 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/data/info.pb 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.15.131 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.3.15.190:34417,127.3.15.189:41941,127.3.15.188:39851 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_enforce_access_control=true 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_inject_invalid_authz_token_ratio=0.5 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --heap_profile_path=/tmp/kudu-tserver.3630 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.3.15.131 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/logs 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-cdh6.x-SNAPSHOT 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5327ca21851e41b4ad88c45c6cffc929a77ae8f4 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type DEBUG 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 06 Apr 2019 03:41:25 PST on quasar-mflytl-1.vpc.cloudera.com 11:18:09.901 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 304881 11:18:09.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.906586 3630 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 41354us 11:18:09.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:09.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.908179 3630 init.cc:359] Logged in from keytab as kudu/127.3.15.131@KRBTEST.COM (short username kudu) 11:18:09.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.909190 3630 fs_manager.cc:263] Metadata directory not provided 11:18:09.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.909363 3630 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 11:18:09.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.909600 3630 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 11:18:09.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.909634 3630 server_base.cc:434] Attempting to create new FS layout instead 11:18:09.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.911321 3630 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/data/instance: 11:18:09.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" 11:18:09.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.911639 3630 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal/instance: 11:18:09.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" 11:18:09.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.916198 3630 fs_manager.cc:503] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s 11:18:09.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.916821 3630 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 11:18:09.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.917237 3630 file_cache.cc:466] Constructed file cache lbm with capacity 419430 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.919896 3630 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.919992 3630 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/data,/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:09 on dist-test-slave-dist-test-slave-6ztd" 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.920126 3630 fs_report.cc:352] FS layout report 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/data/data 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 11:18:09.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 11:18:09.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.953617 3630 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 11:18:09.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.954649 3630 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) 11:18:09.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.954677 3630 ts_tablet_manager.cc:376] Registered 0 tablets 11:18:09.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.961371 3630 rpc_server.cc:205] RPC server started. Bound to: 127.3.15.131:44057 11:18:09.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.961952 3630 webserver.cc:308] Webserver started at http://127.3.15.131:33065/ using document root and password file 11:18:09.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.962370 3630 server_base.cc:608] Dumped server information to /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/data/info.pb 11:18:09.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.966336 3134 external_mini_cluster.cc:995] Started /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver as pid 3630 11:18:09.970 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.131@KRBTEST.COM for kudu/127.3.15.188@KRBTEST.COM 11:18:09.971 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.131@KRBTEST.COM for kudu/127.3.15.189@KRBTEST.COM 11:18:09.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549489, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.15.131@KRBTEST.COM for kudu/127.3.15.190@KRBTEST.COM 11:18:09.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.978436 3751 heartbeater.cc:345] Connected to a master server at 127.3.15.189:41941 11:18:09.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.978452 3750 heartbeater.cc:345] Connected to a master server at 127.3.15.188:39851 11:18:09.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.978531 3750 heartbeater.cc:426] Registering TS with master... 11:18:09.978 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.15.188@KRBTEST.COM 11:18:09.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.978852 3750 heartbeater.cc:475] Master 127.3.15.188:39851 requested a full tablet report, sending... 11:18:09.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.978531 3751 heartbeater.cc:426] Registering TS with master... 11:18:09.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.979257 3751 heartbeater.cc:475] Master 127.3.15.189:41941 requested a full tablet report, sending... 11:18:09.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.979837 3752 heartbeater.cc:345] Connected to a master server at 127.3.15.190:34417 11:18:09.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.979867 3752 heartbeater.cc:426] Registering TS with master... 11:18:09.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.979962 3752 heartbeater.cc:475] Master 127.3.15.190:34417 requested a full tablet report, sending... 11:18:09.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.980273 3310 ts_manager.cc:159] Registered new tserver with Master: 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) 11:18:09.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.980692 3245 ts_manager.cc:159] Registered new tserver with Master: 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) 11:18:09.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.981536 3245 master_service.cc:271] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.15.131@KRBTEST.COM'} at 127.3.15.131:39167 11:18:09.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:09.982409 3177 ts_manager.cc:159] Registered new tserver with Master: 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) 11:18:09.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.15.189@KRBTEST.COM 11:18:09.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:09 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.15.190@KRBTEST.COM 11:18:10.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:10.000578 3134 external_mini_cluster.cc:623] 3 TS(s) registered with all masters 11:18:10.000 [DEBUG - main] (MiniKuduCluster.java:165) Response: 11:18:10.001 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_kdc_env_vars { } 11:18:10.012 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_kdc_env_vars { env_vars { key: "KRB5CCNAME" value: "/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5cc" } env_vars { key: "KUDU_ENABLE_KRB5_REALM_FIX" value: "yes" } env_vars { key: "KRB5_KDC_PROFILE" value: "/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/kdc.conf" } env_vars { key: "KRB5_CONFIG" value: "/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5.conf" } } 11:18:10.014 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_masters { } 11:18:10.026 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.3.15.190" port: 34417 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.3.15.189" port: 41941 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.3.15.188" port: 39851 } } } 11:18:10.030 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_tservers { } 11:18:10.032 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_tservers { tservers { id { type: TSERVER index: 0 } bound_rpc_address { host: "127.3.15.129" port: 33105 } } tservers { id { type: TSERVER index: 1 } bound_rpc_address { host: "127.3.15.130" port: 37203 } } tservers { id { type: TSERVER index: 2 } bound_rpc_address { host: "127.3.15.131" port: 44057 } } } 11:18:10.032 [INFO - main] (KuduTestHarness.java:141) Creating a new Kudu client... 11:18:10.162 [DEBUG - main] (SecurityUtil.java:106) Using ticketCache: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/krb5kdc/krb5cc 11:18:10.179 [DEBUG - main] (SecurityUtil.java:120) Logged in as subject: test-admin@KRBTEST.COM 11:18:10.255 [DEBUG - Thread-5] (NetUtil.java:106) Resolved IP of `127.3.15.188' to /127.3.15.188 in 512149ns 11:18:10.352 [DEBUG - New I/O worker #1] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.188 11:18:10.353 [DEBUG - New I/O worker #2] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.190 11:18:10.356 [DEBUG - New I/O worker #3] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.189 11:18:10.384 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:10.384888 3354 catalog_manager.cc:1111] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b: acquiring CA information for follower catalog manager: success 11:18:10.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:10.386085 3354 catalog_manager.cc:1139] T 00000000000000000000000000000000 P 728d144d58ef4a2a8ec4f57856ae473b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 11:18:10.398 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:10.398715 3352 catalog_manager.cc:1111] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d: acquiring CA information for follower catalog manager: success 11:18:10.400 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:10.400274 3352 catalog_manager.cc:1139] T 00000000000000000000000000000000 P 63cbb0c1927d4d7dbc651859e214889d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 11:18:10.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:10 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.15.190@KRBTEST.COM 11:18:10.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:10 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.15.188@KRBTEST.COM 11:18:10.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:10 dist-test-slave-dist-test-slave-6ztd krb5kdc[3142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.15.189@KRBTEST.COM 11:18:10.484 [DEBUG - New I/O worker #2] (Negotiator.java:764) Authenticated connection [id: 0x09ccc74d, /127.0.0.1:56158 => /127.3.15.190:34417] using SASL/GSSAPI 11:18:10.485 [DEBUG - New I/O worker #1] (Negotiator.java:764) Authenticated connection [id: 0x35504303, /127.0.0.1:37678 => /127.3.15.188:39851] using SASL/GSSAPI 11:18:10.485 [DEBUG - New I/O worker #3] (Negotiator.java:764) Authenticated connection [id: 0xa682caf0, /127.0.0.1:44132 => /127.3.15.189:41941] using SASL/GSSAPI 11:18:10.528 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:10.529 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=278), Traces: [0ms] refreshing cache from master, [34ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [252ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [252ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [258ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, deferred=null) after lookup 11:18:10.532 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:10.533 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=292), Traces: [0ms] refreshing cache from master, [33ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [47ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [55ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [257ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [258ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [282ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=null) after lookup 11:18:10.535 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:10.536 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=289), Traces: [0ms] refreshing cache from master, [34ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [55ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [252ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [258ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [285ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=null) after lookup 11:18:10.537 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:10.538 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=296), Traces: [0ms] refreshing cache from master, [34ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [54ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [253ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [258ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [287ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=null) after lookup 11:18:10.539 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:10.539 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=292), Traces: [0ms] refreshing cache from master, [33ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [55ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [252ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [252ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [288ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=null) after lookup 11:18:10.542 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:10.543 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=292), Traces: [0ms] refreshing cache from master, [33ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [55ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [55ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [252ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [258ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [292ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=null) after lookup 11:18:10.544 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:10.544 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=297), Traces: [0ms] refreshing cache from master, [34ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [55ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [56ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [252ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [259ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [293ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=null) after lookup 11:18:10.545 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:10.546 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=300), Traces: [0ms] refreshing cache from master, [33ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [56ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [56ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [253ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [259ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [295ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=null) after lookup 11:18:10.557 [DEBUG - New I/O worker #1] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:10.557 [DEBUG - New I/O worker #2] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:10.559 [DEBUG - New I/O worker #3] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:10.559 [INFO - New I/O worker #3] (AsyncKuduClient.java:2058) Invalidating location master-127.3.15.189:41941(127.3.15.189:41941) for tablet Kudu Master: connection disconnected 11:18:10.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:10.559531 3227 connection.cc:320] Error closing socket: Network error: TlsSocket::Close: Connection reset by peer 11:18:10.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:10.647230 3484 heartbeater.cc:467] Master 127.3.15.189:41941 was elected leader, sending a full tablet report... 11:18:10.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:10.649380 3214 fault_injection.cc:44] FAULT INJECTION ENABLED! 11:18:10.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:10.649412 3214 fault_injection.cc:45] THIS SERVER MAY CRASH! 11:18:10.652 [DEBUG - New I/O worker #5] (Negotiator.java:764) Authenticated connection [id: 0x9b37e1d5, /127.0.0.1:56164 => /127.3.15.190:34417] using TOKEN/null 11:18:10.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:10.654438 3281 fault_injection.cc:44] FAULT INJECTION ENABLED! 11:18:10.654 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:10.654466 3281 fault_injection.cc:45] THIS SERVER MAY CRASH! 11:18:10.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:10.654794 3281 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44138: invalid token 11:18:10.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:10.654930 3758 fault_injection.cc:44] FAULT INJECTION ENABLED! 11:18:10.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:10.654954 3758 fault_injection.cc:45] THIS SERVER MAY CRASH! 11:18:10.656 [DEBUG - New I/O worker #4] (Negotiator.java:764) Authenticated connection [id: 0x3fb7dc37, /127.0.0.1:37684 => /127.3.15.188:39851] using TOKEN/null 11:18:10.659 [DEBUG - New I/O worker #6] (Negotiator.java:291) peer /127.3.15.189:41941 sent connection negotiation error: Not authorized: invalid token 11:18:10.660 [DEBUG - New I/O worker #6] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:10.663 [DEBUG - New I/O worker #7] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.188 11:18:10.664 [DEBUG - New I/O worker #8] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.190 11:18:10.664 [DEBUG - New I/O worker #1] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.189 11:18:10.676 [DEBUG - New I/O worker #1] (Negotiator.java:764) Authenticated connection [id: 0x2b38b24a, /127.0.0.1:44144 => /127.3.15.189:41941] using SASL/GSSAPI 11:18:10.676 [DEBUG - New I/O worker #8] (Negotiator.java:764) Authenticated connection [id: 0xddd9b6f7, /127.0.0.1:56170 => /127.3.15.190:34417] using SASL/GSSAPI 11:18:10.677 [DEBUG - New I/O worker #7] (Negotiator.java:764) Authenticated connection [id: 0xc8932ce1, /127.0.0.1:37690 => /127.3.15.188:39851] using SASL/GSSAPI 11:18:10.678 [INFO - New I/O worker #1] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:10.681 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:10.681 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ConnectToMaster, tablet=null, attempt=2, TimeoutTracker(timeout=49680, elapsed=114), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [94ms] waiting for new authn token, [112ms] refreshing cache from master, [112ms] sending RPC to server master-127.3.15.188:39851, [113ms] sending RPC to server master-127.3.15.190:34417, [113ms] sending RPC to server master-127.3.15.189:41941, [114ms] received response from server master-127.3.15.188:39851: OK, [114ms] received response from server master-127.3.15.190:34417: OK, [114ms] received response from server master-127.3.15.189:41941: OK, deferred=Deferred@1571718869(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.3.15.189:41941, errback=org.apache.kudu.client.ConnectToCluster$1@360cf2d3 -> ConnectToMasterErrCB for 127.3.15.189:41941)) after lookup 11:18:10.683 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:10.684 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTabletServers, tablet=Kudu Master, attempt=3, TimeoutTracker(timeout=50000, elapsed=437), Traces: [0ms] refreshing cache from master, [33ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [56ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [56ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [253ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [259ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [295ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [296ms] sending RPC to server master-127.3.15.189:41941, [309ms] received response from server master-127.3.15.189:41941: Network error: connection disconnected, [309ms] delaying RPC due to: Network error: connection disconnected, [315ms] refreshing cache from master, [316ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [316ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [316ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [405ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [407ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [410ms] Sub RPC ConnectToMaster: waiting for new authn token, [428ms] Sub RPC ConnectToMaster: refreshing cache from master, [428ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [429ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [429ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [430ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [430ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [430ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [431ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [433ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=Deferred@1693049214(state=PENDING, result=null, callback=(continuation of Deferred@2085529398 after retry RPC@1442778692), errback=(continuation of Deferred@2085529398 after retry RPC@1442778692))) after lookup 11:18:10.686 [DEBUG - New I/O worker #7] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:10.686 [DEBUG - New I/O worker #5] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:10.686 [DEBUG - New I/O worker #4] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:10.687 [DEBUG - New I/O worker #8] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:10.687 [DEBUG - New I/O worker #1] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:10.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:10.830130 3622 heartbeater.cc:467] Master 127.3.15.189:41941 was elected leader, sending a full tablet report... 11:18:10.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:10.983063 3751 heartbeater.cc:467] Master 127.3.15.189:41941 was elected leader, sending a full tablet report... 11:18:12.571 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:12.571425 3777 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44146: token expired 11:18:12.572 [DEBUG - New I/O worker #2] (Negotiator.java:291) peer /127.3.15.189:41941 sent connection negotiation error: Not authorized: token expired 11:18:12.572 [DEBUG - New I/O worker #2] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:12.585 [DEBUG - New I/O worker #3] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.188 11:18:12.585 [DEBUG - New I/O worker #4] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.190 11:18:12.597 [DEBUG - New I/O worker #5] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.189 11:18:12.609 [DEBUG - New I/O worker #4] (Negotiator.java:764) Authenticated connection [id: 0xd5236482, /127.0.0.1:56178 => /127.3.15.190:34417] using SASL/GSSAPI 11:18:12.613 [DEBUG - New I/O worker #3] (Negotiator.java:764) Authenticated connection [id: 0xe0b418a1, /127.0.0.1:37698 => /127.3.15.188:39851] using SASL/GSSAPI 11:18:12.621 [DEBUG - New I/O worker #5] (Negotiator.java:764) Authenticated connection [id: 0xca06c522, /127.0.0.1:44152 => /127.3.15.189:41941] using SASL/GSSAPI 11:18:12.623 [INFO - New I/O worker #5] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:12.641 [DEBUG - New I/O worker #3] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:12.643 [DEBUG - New I/O worker #4] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:12.645 [DEBUG - New I/O worker #5] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer 11:18:12.645 [INFO - New I/O worker #5] (AsyncKuduClient.java:2058) Invalidating location master-127.3.15.189:41941(127.3.15.189:41941) for tablet Kudu Master: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer 11:18:12.645 [INFO - New I/O worker #5] (AsyncKuduClient.java:2058) Invalidating location master-127.3.15.189:41941(127.3.15.189:41941) for tablet Kudu Master: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer 11:18:12.646 [DEBUG - New I/O worker #5] (RemoteTablet.java:131) tablet Kudu Master already removed ts master-127.3.15.189:41941, size left is 0 11:18:12.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:12.646572 3229 connection.cc:647] server connection from 127.0.0.1:44152 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44152): Connection reset by peer (error 104) 11:18:12.674 [DEBUG - New I/O worker #7] (Negotiator.java:764) Authenticated connection [id: 0x883d7b6c, /127.0.0.1:56184 => /127.3.15.190:34417] using TOKEN/null 11:18:12.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:12.675473 3778 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37704: invalid token signature 11:18:12.676 [DEBUG - New I/O worker #6] (Negotiator.java:291) peer /127.3.15.188:39851 sent connection negotiation error: Not authorized: invalid token signature 11:18:12.678 [DEBUG - New I/O worker #6] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:12.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:12.680547 3777 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44158: token expired 11:18:12.682 [DEBUG - New I/O worker #1] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.188 11:18:12.684 [DEBUG - New I/O worker #8] (Negotiator.java:291) peer /127.3.15.189:41941 sent connection negotiation error: Not authorized: token expired 11:18:12.685 [DEBUG - New I/O worker #8] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:12.689 [DEBUG - New I/O worker #2] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.190 11:18:12.692 [DEBUG - New I/O worker #3] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.189 11:18:12.705 [DEBUG - New I/O worker #3] (Negotiator.java:764) Authenticated connection [id: 0xf8e34d98, /127.0.0.1:44164 => /127.3.15.189:41941] using SASL/GSSAPI 11:18:12.707 [DEBUG - New I/O worker #2] (Negotiator.java:764) Authenticated connection [id: 0x9611cf0a, /127.0.0.1:56190 => /127.3.15.190:34417] using SASL/GSSAPI 11:18:12.708 [DEBUG - New I/O worker #1] (Negotiator.java:764) Authenticated connection [id: 0x0c882b52, /127.0.0.1:37710 => /127.3.15.188:39851] using SASL/GSSAPI 11:18:12.717 [INFO - New I/O worker #3] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:12.733 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.734 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTables, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=48), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [5ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [6ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [30ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [47ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=null) after lookup 11:18:12.736 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.737 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ConnectToMaster, tablet=null, attempt=2, TimeoutTracker(timeout=49892, elapsed=70), Traces: [0ms] sending RPC to server master-127.3.15.188:39851, [13ms] waiting for new authn token, [51ms] refreshing cache from master, [51ms] sending RPC to server master-127.3.15.188:39851, [52ms] sending RPC to server master-127.3.15.190:34417, [52ms] sending RPC to server master-127.3.15.189:41941, [54ms] received response from server master-127.3.15.188:39851: OK, [55ms] received response from server master-127.3.15.190:34417: OK, [70ms] received response from server master-127.3.15.189:41941: OK, deferred=Deferred@1164605071(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.3.15.188:39851, errback=org.apache.kudu.client.ConnectToCluster$1@56e0cd68 -> ConnectToMasterErrCB for 127.3.15.188:39851)) after lookup 11:18:12.738 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.738 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ConnectToMaster, tablet=null, attempt=2, TimeoutTracker(timeout=49892, elapsed=71), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [19ms] waiting for new authn token, [53ms] refreshing cache from master, [53ms] sending RPC to server master-127.3.15.188:39851, [53ms] sending RPC to server master-127.3.15.190:34417, [55ms] received response from server master-127.3.15.188:39851: OK, [58ms] sending RPC to server master-127.3.15.189:41941, [59ms] received response from server master-127.3.15.190:34417: OK, [71ms] received response from server master-127.3.15.189:41941: OK, deferred=Deferred@707920792(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.3.15.189:41941, errback=org.apache.kudu.client.ConnectToCluster$1@8f03661 -> ConnectToMasterErrCB for 127.3.15.189:41941)) after lookup 11:18:12.739 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.740 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ConnectToMaster, tablet=null, attempt=2, TimeoutTracker(timeout=49892, elapsed=72), Traces: [0ms] sending RPC to server master-127.3.15.188:39851, [19ms] waiting for new authn token, [57ms] refreshing cache from master, [57ms] sending RPC to server master-127.3.15.188:39851, [57ms] sending RPC to server master-127.3.15.190:34417, [58ms] sending RPC to server master-127.3.15.189:41941, [60ms] received response from server master-127.3.15.190:34417: OK, [62ms] received response from server master-127.3.15.188:39851: OK, [72ms] received response from server master-127.3.15.189:41941: OK, deferred=Deferred@515438590(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.3.15.188:39851, errback=org.apache.kudu.client.ConnectToCluster$1@4e63f2cc -> ConnectToMasterErrCB for 127.3.15.188:39851)) after lookup 11:18:12.741 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.742 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ConnectToMaster, tablet=null, attempt=2, TimeoutTracker(timeout=49892, elapsed=74), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [20ms] waiting for new authn token, [60ms] refreshing cache from master, [61ms] sending RPC to server master-127.3.15.188:39851, [61ms] sending RPC to server master-127.3.15.190:34417, [62ms] received response from server master-127.3.15.188:39851: OK, [65ms] sending RPC to server master-127.3.15.189:41941, [66ms] received response from server master-127.3.15.190:34417: OK, [73ms] received response from server master-127.3.15.189:41941: OK, deferred=Deferred@618883796(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.3.15.189:41941, errback=org.apache.kudu.client.ConnectToCluster$1@75471916 -> ConnectToMasterErrCB for 127.3.15.189:41941)) after lookup 11:18:12.748 [DEBUG - New I/O worker #1] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:12.749 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.751 [DEBUG - New I/O worker #7] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:12.750 [DEBUG - New I/O worker #2] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:12.753 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTables, tablet=Kudu Master, attempt=3, TimeoutTracker(timeout=50000, elapsed=194), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [26ms] waiting for new authn token, [68ms] sending RPC to server master-127.3.15.189:41941, [88ms] received response from server master-127.3.15.189:41941: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [88ms] delaying RPC due to: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [108ms] refreshing cache from master, [109ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [109ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [109ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [121ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [122ms] Sub RPC ConnectToMaster: waiting for new authn token, [128ms] Sub RPC ConnectToMaster: waiting for new authn token, [160ms] Sub RPC ConnectToMaster: refreshing cache from master, [160ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [162ms] Sub RPC ConnectToMaster: refreshing cache from master, [162ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [162ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [163ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [164ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [164ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [167ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [168ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [179ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [180ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [180ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [191ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=Deferred@801714109(state=PENDING, result=null, callback=wakeup thread Thread-2, errback=wakeup thread Thread-2)) after lookup 11:18:12.761 [DEBUG - New I/O worker #4] (Negotiator.java:764) Authenticated connection [id: 0x3ffc99b4, /127.0.0.1:37716 => /127.3.15.188:39851] using TOKEN/null 11:18:12.769 [DEBUG - New I/O worker #4] (RemoteTablet.java:153) master-127.3.15.188:39851 was demoted as the leader for Kudu Master 11:18:12.797 [DEBUG - New I/O worker #5] (Negotiator.java:764) Authenticated connection [id: 0xed6a9ee2, /127.0.0.1:56196 => /127.3.15.190:34417] using TOKEN/null 11:18:12.802 [DEBUG - New I/O worker #3] (ConnectToCluster.java:331) Callback already invoked, discarding response(role: LEADER ca_cert_der: "0\202\001\3170\202\0018\240\003\002\001\002\002\t\000\256\311\337\365z\322\26540\r\006\t*\206H\206\367\r\001\001\v\005\0000\0271\0250\023\006\003U\004\003\023\fkudu-ipki-ca0\036\027\r190406111809Z\027\r290403111809Z0\0271\0250\023\006\003U\004\003\023\fkudu-ipki-ca0\201\2370\r\006\t*\206H\206\367\r\001\001\001\005\000\003\201\215\0000\201\211\002\201\201\000\275<\341\223\304\316\333D\353\323\334\004\346;\372\323i\3144\206\024\214dP\2272/4\327k\214\362)\004\363\254u\001\235\230\tA\005\224\367\222\035\003\213\244\371\241\031T4\264\321\325\376*\327\217_\3571{\221\301\223\036J\341m=\3363\251q\21535r\030\253Vh}9:\256\200\246\225\212\220\230\366\302X\317\301\036\202Ey0\341\334\232b\252\355\231{T\316\317\2475\237\267W\f\344o\216\336\261\002\003\001\000\001\243#0!0\016\006\003U\035\017\001\001\377\004\004\003\002\002\0040\017\006\003U\035\023\001\001\377\004\0050\003\001\001\3770\r\006\t*\206H\206\367\r\001\001\v\005\000\003\201\201\000\213\206\305\253\355,\2264\r1\353\002KN\376\341\216Rk\310\324U_\247\325\305\2067\3016J\313\366\376\336\006\246\376\274wr\323\316>!\003\021+\324xl\316OG#\022\017\364L\314\361<\037U\364\353\333\ff\216\001\222\353\300C\326\265\275\274\224\256\246~+H\211+KH|\347\270J\374\342\247\245\024\335\357\233V\203N\230\362\016\031\310\342\277\270\340S\262\234R[}\376\340R\356V\366@\304(" authn_token { token_data: "\b\365\225\242\345\005\032\f\n\ntest-admin" signature: "\032i5\247n\363y\265%t#\001e;\216\203t U\342j1\254Y\205]\332\310\2447sy?\275V\t[o-^\222\360\370_1\000W\377T\237R\207\305tVzrS\030\350%jG\324" signing_key_seq_num: 0 } master_addrs { host: "127.3.15.190" port: 34417 } master_addrs { host: "127.3.15.189" port: 41941 } master_addrs { host: "127.3.15.188" port: 39851 } ) from 127.3.15.189:41941 11:18:12.802 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.804 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTables, tablet=Kudu Master, attempt=3, TimeoutTracker(timeout=50000, elapsed=243), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [24ms] waiting for new authn token, [66ms] sending RPC to server master-127.3.15.189:41941, [86ms] received response from server master-127.3.15.189:41941: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [87ms] delaying RPC due to: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [108ms] refreshing cache from master, [108ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [108ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [108ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [122ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [127ms] Sub RPC ConnectToMaster: waiting for new authn token, [128ms] Sub RPC ConnectToMaster: waiting for new authn token, [165ms] Sub RPC ConnectToMaster: refreshing cache from master, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [166ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [168ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [168ms] Sub RPC ConnectToMaster: refreshing cache from master, [169ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [169ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [170ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [170ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [173ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [174ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [180ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [181ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [183ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [243ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=Deferred@1480787354(state=PENDING, result=null, callback=wakeup thread Thread-7, errback=wakeup thread Thread-7)) after lookup 11:18:12.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:12.805852 3228 connection.cc:320] Error closing socket: Network error: TlsSocket::Close: Connection reset by peer 11:18:12.809 [DEBUG - New I/O worker #3] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:12.808 [DEBUG - New I/O worker #4] (RemoteTablet.java:153) master-127.3.15.188:39851 was demoted as the leader for Kudu Master 11:18:12.809 [INFO - New I/O worker #3] (AsyncKuduClient.java:2058) Invalidating location master-127.3.15.189:41941(127.3.15.189:41941) for tablet Kudu Master: connection disconnected 11:18:12.810 [INFO - New I/O worker #3] (ConnectToCluster.java:361) Unable to connect to master 127.3.15.189:41941: connection disconnected 11:18:12.813 [WARN - New I/O worker #3] (ConnectToCluster.java:281) Unable to find the leader master 127.3.15.188:39851,127.3.15.190:34417,127.3.15.189:41941; will retry 11:18:12.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:12.837487 3777 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44170: invalid token 11:18:12.837 [DEBUG - New I/O worker #6] (Negotiator.java:291) peer /127.3.15.189:41941 sent connection negotiation error: Not authorized: invalid token 11:18:12.841 [DEBUG - New I/O worker #6] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:12.845 [DEBUG - New I/O worker #7] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.188 11:18:12.847 [DEBUG - New I/O worker #1] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.189 11:18:12.849 [DEBUG - New I/O worker #8] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.190 11:18:12.862 [DEBUG - New I/O worker #1] (Negotiator.java:764) Authenticated connection [id: 0x68f05618, /127.0.0.1:44176 => /127.3.15.189:41941] using SASL/GSSAPI 11:18:12.865 [DEBUG - New I/O worker #7] (Negotiator.java:764) Authenticated connection [id: 0x0530c01b, /127.0.0.1:37722 => /127.3.15.188:39851] using SASL/GSSAPI 11:18:12.865 [DEBUG - New I/O worker #8] (Negotiator.java:764) Authenticated connection [id: 0x39c412bb, /127.0.0.1:56202 => /127.3.15.190:34417] using SASL/GSSAPI 11:18:12.868 [INFO - New I/O worker #1] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:12.881 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.882 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ConnectToMaster, tablet=null, attempt=2, TimeoutTracker(timeout=49733, elapsed=55), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [16ms] waiting for new authn token, [43ms] refreshing cache from master, [43ms] sending RPC to server master-127.3.15.188:39851, [46ms] received response from server master-127.3.15.188:39851: OK, [50ms] sending RPC to server master-127.3.15.190:34417, [51ms] sending RPC to server master-127.3.15.189:41941, [53ms] received response from server master-127.3.15.190:34417: OK, [55ms] received response from server master-127.3.15.189:41941: OK, deferred=Deferred@186897887(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.3.15.189:41941, errback=org.apache.kudu.client.ConnectToCluster$1@2816260d -> ConnectToMasterErrCB for 127.3.15.189:41941)) after lookup 11:18:12.883 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.883 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ConnectToMaster, tablet=null, attempt=2, TimeoutTracker(timeout=49733, elapsed=55), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [17ms] waiting for new authn token, [50ms] refreshing cache from master, [50ms] sending RPC to server master-127.3.15.188:39851, [51ms] sending RPC to server master-127.3.15.190:34417, [51ms] sending RPC to server master-127.3.15.189:41941, [52ms] received response from server master-127.3.15.188:39851: OK, [53ms] received response from server master-127.3.15.190:34417: OK, [56ms] received response from server master-127.3.15.189:41941: OK, deferred=Deferred@1871785211(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.3.15.189:41941, errback=org.apache.kudu.client.ConnectToCluster$1@52bcfb26 -> ConnectToMasterErrCB for 127.3.15.189:41941)) after lookup 11:18:12.889 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.889 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ConnectToMaster, tablet=null, attempt=2, TimeoutTracker(timeout=49729, elapsed=60), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [19ms] waiting for new authn token, [50ms] refreshing cache from master, [50ms] sending RPC to server master-127.3.15.188:39851, [52ms] sending RPC to server master-127.3.15.190:34417, [52ms] sending RPC to server master-127.3.15.189:41941, [53ms] received response from server master-127.3.15.188:39851: OK, [54ms] received response from server master-127.3.15.190:34417: OK, [60ms] received response from server master-127.3.15.189:41941: OK, deferred=Deferred@1201947011(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.3.15.189:41941, errback=org.apache.kudu.client.ConnectToCluster$1@5e9b2134 -> ConnectToMasterErrCB for 127.3.15.189:41941)) after lookup 11:18:12.890 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.892 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTables, tablet=Kudu Master, attempt=5, TimeoutTracker(timeout=50000, elapsed=331), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [24ms] waiting for new authn token, [66ms] sending RPC to server master-127.3.15.189:41941, [86ms] received response from server master-127.3.15.189:41941: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [87ms] delaying RPC due to: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [108ms] refreshing cache from master, [108ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [108ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [108ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [122ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [127ms] Sub RPC ConnectToMaster: waiting for new authn token, [128ms] Sub RPC ConnectToMaster: waiting for new authn token, [165ms] Sub RPC ConnectToMaster: refreshing cache from master, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [165ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [166ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [168ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [168ms] Sub RPC ConnectToMaster: refreshing cache from master, [169ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [169ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [170ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [170ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [173ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [174ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [180ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [181ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [183ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [243ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [245ms] sending RPC to server master-127.3.15.188:39851, [250ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: Network error: connection disconnected, [250ms] delaying RPC due to: Illegal state: Not the leader. Local UUID: 728d144d58ef4a2a8ec4f57856ae473b, Raft Consensus state: current_term: 1 leader_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } (error 0), [250ms] received response from server master-127.3.15.188:39851: Illegal state: Not the leader. Local UUID: 728d144d58ef4a2a8ec4f57856ae473b, Raft Consensus state: current_term: 1 leader_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } (error 0), [250ms] Sub RPC ConnectToMaster: delaying RPC due to: Network error: connection disconnected, [266ms] refreshing cache from master, [266ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [266ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [267ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [267ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [268ms] Sub RPC ConnectToMaster: refreshing cache from master, [268ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [268ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [268ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [268ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [270ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [271ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [283ms] Sub RPC ConnectToMaster: waiting for new authn token, [285ms] Sub RPC ConnectToMaster: waiting for new authn token, [310ms] Sub RPC ConnectToMaster: refreshing cache from master, [310ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [313ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [317ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [318ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [318ms] Sub RPC ConnectToMaster: refreshing cache from master, [318ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [319ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [319ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [320ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [320ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [321ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [322ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [323ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [324ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [325ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [331ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=Deferred@1480787354(state=PENDING, result=null, callback=wakeup thread Thread-7 -> (continuation of Deferred@948644072 after retry RPC@232493332), errback=wakeup thread Thread-7 -> (continuation of Deferred@948644072 after retry RPC@232493332))) after lookup 11:18:12.893 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.894 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ConnectToMaster, tablet=Kudu Master, attempt=4, TimeoutTracker(timeout=49892, elapsed=226), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [20ms] waiting for new authn token, [60ms] refreshing cache from master, [61ms] sending RPC to server master-127.3.15.188:39851, [61ms] sending RPC to server master-127.3.15.190:34417, [62ms] received response from server master-127.3.15.188:39851: OK, [65ms] sending RPC to server master-127.3.15.189:41941, [66ms] received response from server master-127.3.15.190:34417: OK, [73ms] received response from server master-127.3.15.189:41941: OK, [75ms] sending RPC to server master-127.3.15.189:41941, [142ms] received response from server master-127.3.15.189:41941: Network error: connection disconnected, [142ms] delaying RPC due to: Network error: connection disconnected, [160ms] refreshing cache from master, [160ms] sending RPC to server master-127.3.15.188:39851, [160ms] sending RPC to server master-127.3.15.190:34417, [160ms] sending RPC to server master-127.3.15.189:41941, [162ms] received response from server master-127.3.15.188:39851: OK, [163ms] received response from server master-127.3.15.190:34417: OK, [177ms] waiting for new authn token, [210ms] refreshing cache from master, [210ms] sending RPC to server master-127.3.15.188:39851, [211ms] sending RPC to server master-127.3.15.190:34417, [211ms] sending RPC to server master-127.3.15.189:41941, [212ms] received response from server master-127.3.15.188:39851: OK, [213ms] received response from server master-127.3.15.190:34417: OK, [216ms] received response from server master-127.3.15.189:41941: OK, [217ms] sending RPC to server master-127.3.15.189:41941, [226ms] received response from server master-127.3.15.189:41941: OK, deferred=Deferred@618883796(state=PENDING, result=null, callback=passthrough -> ConnectToMasterCB for 127.3.15.189:41941 -> (continuation of Deferred@1746301301 after retry RPC@1862500290), errback=org.apache.kudu.client.ConnectToCluster$1@75471916 -> ConnectToMasterErrCB for 127.3.15.189:41941 -> (continuation of Deferred@1746301301 after retry RPC@1862500290))) after lookup 11:18:12.895 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:12.896 [DEBUG - New I/O worker #1] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=ListTables, tablet=Kudu Master, attempt=6, TimeoutTracker(timeout=50000, elapsed=337), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [26ms] waiting for new authn token, [68ms] sending RPC to server master-127.3.15.189:41941, [88ms] received response from server master-127.3.15.189:41941: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [88ms] delaying RPC due to: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [108ms] refreshing cache from master, [109ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [109ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [109ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [121ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [122ms] Sub RPC ConnectToMaster: waiting for new authn token, [128ms] Sub RPC ConnectToMaster: waiting for new authn token, [160ms] Sub RPC ConnectToMaster: refreshing cache from master, [160ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [162ms] Sub RPC ConnectToMaster: refreshing cache from master, [162ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [162ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [163ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [164ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [164ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [167ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [168ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [179ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [180ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [180ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [191ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [196ms] sending RPC to server master-127.3.15.188:39851, [197ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [213ms] delaying RPC due to: Illegal state: Not the leader. Local UUID: 728d144d58ef4a2a8ec4f57856ae473b, Raft Consensus state: current_term: 1 leader_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } (error 0), [213ms] received response from server master-127.3.15.188:39851: Illegal state: Not the leader. Local UUID: 728d144d58ef4a2a8ec4f57856ae473b, Raft Consensus state: current_term: 1 leader_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63cbb0c1927d4d7dbc651859e214889d" member_type: VOTER last_known_addr { host: "127.3.15.190" port: 34417 } } peers { permanent_uuid: "a04f897cb6ac4fe38cfe92cc0220fa7c" member_type: VOTER last_known_addr { host: "127.3.15.189" port: 41941 } } peers { permanent_uuid: "728d144d58ef4a2a8ec4f57856ae473b" member_type: VOTER last_known_addr { host: "127.3.15.188" port: 39851 } } } (error 0), [228ms] refreshing cache from master, [228ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [230ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [230ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [234ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [242ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [252ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: Network error: connection disconnected, [256ms] delaying RPC due to: Service unavailable: Master config (127.3.15.188:39851,127.3.15.190:34417,127.3.15.189:41941) has no leader. Exceptions received: org.apache.kudu.client.RecoverableException: connection disconnected, [270ms] refreshing cache from master, [271ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [271ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [271ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [273ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [274ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [290ms] Sub RPC ConnectToMaster: waiting for new authn token, [321ms] Sub RPC ConnectToMaster: refreshing cache from master, [321ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [323ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [323ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [324ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [325ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [331ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [332ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [337ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=Deferred@801714109(state=PENDING, result=null, callback=wakeup thread Thread-2 -> (continuation of Deferred@237950626 after retry RPC@1596761410) -> (continuation of Deferred@82508299 after retry RPC after error@1885567035), errback=wakeup thread Thread-2 -> (continuation of Deferred@237950626 after retry RPC@1596761410) -> (continuation of Deferred@82508299 after retry RPC after error@1885567035))) after lookup 11:18:12.898 [DEBUG - New I/O worker #7] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:12.904 [DEBUG - New I/O worker #4] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:12.906 [DEBUG - New I/O worker #1] (ConnectToCluster.java:331) Callback already invoked, discarding response(role: LEADER ca_cert_der: "0\202\001\3170\202\0018\240\003\002\001\002\002\t\000\256\311\337\365z\322\26540\r\006\t*\206H\206\367\r\001\001\v\005\0000\0271\0250\023\006\003U\004\003\023\fkudu-ipki-ca0\036\027\r190406111809Z\027\r290403111809Z0\0271\0250\023\006\003U\004\003\023\fkudu-ipki-ca0\201\2370\r\006\t*\206H\206\367\r\001\001\001\005\000\003\201\215\0000\201\211\002\201\201\000\275<\341\223\304\316\333D\353\323\334\004\346;\372\323i\3144\206\024\214dP\2272/4\327k\214\362)\004\363\254u\001\235\230\tA\005\224\367\222\035\003\213\244\371\241\031T4\264\321\325\376*\327\217_\3571{\221\301\223\036J\341m=\3363\251q\21535r\030\253Vh}9:\256\200\246\225\212\220\230\366\302X\317\301\036\202Ey0\341\334\232b\252\355\231{T\316\317\2475\237\267W\f\344o\216\336\261\002\003\001\000\001\243#0!0\016\006\003U\035\017\001\001\377\004\004\003\002\002\0040\017\006\003U\035\023\001\001\377\004\0050\003\001\001\3770\r\006\t*\206H\206\367\r\001\001\v\005\000\003\201\201\000\213\206\305\253\355,\2264\r1\353\002KN\376\341\216Rk\310\324U_\247\325\305\2067\3016J\313\366\376\336\006\246\376\274wr\323\316>!\003\021+\324xl\316OG#\022\017\364L\314\361<\037U\364\353\333\ff\216\001\222\353\300C\326\265\275\274\224\256\246~+H\211+KH|\347\270J\374\342\247\245\024\335\357\233V\203N\230\362\016\031\310\342\277\270\340S\262\234R[}\376\340R\356V\366@\304(" authn_token { token_data: "\b\365\225\242\345\005\032\f\n\ntest-admin" signature: "\032i5\247n\363y\265%t#\001e;\216\203t U\342j1\254Y\205]\332\310\2447sy?\275V\t[o-^\222\360\370_1\000W\377T\237R\207\305tVzrS\030\350%jG\324" signing_key_seq_num: 0 } master_addrs { host: "127.3.15.190" port: 34417 } master_addrs { host: "127.3.15.189" port: 41941 } master_addrs { host: "127.3.15.188" port: 39851 } ) from 127.3.15.189:41941 11:18:12.906 [DEBUG - New I/O worker #8] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:12.907 [DEBUG - New I/O worker #5] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:12.911 [DEBUG - New I/O worker #1] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:14.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.675720 3780 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44178: token expired 11:18:14.676 [DEBUG - New I/O worker #2] (Negotiator.java:291) peer /127.3.15.189:41941 sent connection negotiation error: Not authorized: token expired 11:18:14.676 [DEBUG - New I/O worker #2] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:14.681 [DEBUG - New I/O worker #3] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.188 11:18:14.681 [DEBUG - New I/O worker #4] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.190 11:18:14.685 [DEBUG - New I/O worker #5] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.189 11:18:14.694 [DEBUG - New I/O worker #4] (Negotiator.java:764) Authenticated connection [id: 0x4b0b12f9, /127.0.0.1:56210 => /127.3.15.190:34417] using SASL/GSSAPI 11:18:14.696 [DEBUG - New I/O worker #5] (Negotiator.java:764) Authenticated connection [id: 0xde1464d6, /127.0.0.1:44184 => /127.3.15.189:41941] using SASL/GSSAPI 11:18:14.700 [DEBUG - New I/O worker #3] (Negotiator.java:764) Authenticated connection [id: 0x7d4a3e50, /127.0.0.1:37730 => /127.3.15.188:39851] using SASL/GSSAPI 11:18:14.701 [INFO - New I/O worker #5] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:14.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.704560 3244 catalog_manager.cc:1415] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44184: 11:18:14.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "TestAuthTokenReacquire-table-1" 11:18:14.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 11:18:14.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:14.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 11:18:14.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 11:18:14.705 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.705137 3242 catalog_manager.cc:1415] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44184: 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "TestAuthTokenReacquire-table-6" 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 11:18:14.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.705343 3240 catalog_manager.cc:1415] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44184: 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "TestAuthTokenReacquire-table-7" 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.705365 3245 catalog_manager.cc:1415] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44184: 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "TestAuthTokenReacquire-table-4" 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.705518 3243 catalog_manager.cc:1415] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44184: 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "TestAuthTokenReacquire-table-3" 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.706817 3244 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAuthTokenReacquire-table-1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 11:18:14.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.706862 3242 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAuthTokenReacquire-table-6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 11:18:14.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.707203 3243 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAuthTokenReacquire-table-3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 11:18:14.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.707417 3240 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAuthTokenReacquire-table-7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 11:18:14.712 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.708431 3245 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAuthTokenReacquire-table-4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 11:18:14.739 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.734349 3687 tablet_service.cc:1006] Processing CreateTablet for tablet 1734b9064c4943339961de5d872e236b (table=TestAuthTokenReacquire-table-1 [id=3aa15bdd6f364b75a22e2b2be21a8a01]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.734825 3687 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1734b9064c4943339961de5d872e236b. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.735854 3558 tablet_service.cc:1006] Processing CreateTablet for tablet 1734b9064c4943339961de5d872e236b (table=TestAuthTokenReacquire-table-1 [id=3aa15bdd6f364b75a22e2b2be21a8a01]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.736290 3558 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1734b9064c4943339961de5d872e236b. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.737512 3557 tablet_service.cc:1006] Processing CreateTablet for tablet fe1a243ef4c64777b0a73ede1d7f653a (table=TestAuthTokenReacquire-table-4 [id=329758112d6c49beb826456c5e22abca]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.737676 3557 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fe1a243ef4c64777b0a73ede1d7f653a. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.740 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.739683 3807 tablet_bootstrap.cc:439] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd: Bootstrap starting. 11:18:14.760 [DEBUG - New I/O worker #5] (AsyncKuduClient.java:791) Opened table 3aa15bdd6f364b75a22e2b2be21a8a01 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.740831 3556 tablet_service.cc:1006] Processing CreateTablet for tablet 3337cdf76ae447bd9df561914cede569 (table=TestAuthTokenReacquire-table-7 [id=d6d879c7542741c8a7be7786b888ddd3]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.740969 3556 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 3337cdf76ae447bd9df561914cede569. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.742224 3807 tablet_bootstrap.cc:590] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd: No blocks or log segments found. Creating new log. 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.742269 3555 tablet_service.cc:1006] Processing CreateTablet for tablet c07b620efb024973aea4f313eabf4592 (table=TestAuthTokenReacquire-table-6 [id=d70654235532446a85c0dc05f79f3c4f]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.742391 3555 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet c07b620efb024973aea4f313eabf4592. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.742895 3807 log.cc:525] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd: Log is configured to *not* fsync() on all Append() calls 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.743424 3554 tablet_service.cc:1006] Processing CreateTablet for tablet c156fd8d4e8f436c97eed32d4b50e1cd (table=TestAuthTokenReacquire-table-3 [id=a45d2c9dd7fe4a04a35d322134f2a9fa]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.743542 3554 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet c156fd8d4e8f436c97eed32d4b50e1cd. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.743764 3420 tablet_service.cc:1006] Processing CreateTablet for tablet 1734b9064c4943339961de5d872e236b (table=TestAuthTokenReacquire-table-1 [id=3aa15bdd6f364b75a22e2b2be21a8a01]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.744191 3420 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1734b9064c4943339961de5d872e236b. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.744870 3419 tablet_service.cc:1006] Processing CreateTablet for tablet fe1a243ef4c64777b0a73ede1d7f653a (table=TestAuthTokenReacquire-table-4 [id=329758112d6c49beb826456c5e22abca]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.745064 3419 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fe1a243ef4c64777b0a73ede1d7f653a. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.745174 3686 tablet_service.cc:1006] Processing CreateTablet for tablet 3337cdf76ae447bd9df561914cede569 (table=TestAuthTokenReacquire-table-7 [id=d6d879c7542741c8a7be7786b888ddd3]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.745324 3686 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 3337cdf76ae447bd9df561914cede569. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.745354 3683 tablet_service.cc:1006] Processing CreateTablet for tablet c07b620efb024973aea4f313eabf4592 (table=TestAuthTokenReacquire-table-6 [id=d70654235532446a85c0dc05f79f3c4f]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.745491 3683 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet c07b620efb024973aea4f313eabf4592. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.746716 3685 tablet_service.cc:1006] Processing CreateTablet for tablet c156fd8d4e8f436c97eed32d4b50e1cd (table=TestAuthTokenReacquire-table-3 [id=a45d2c9dd7fe4a04a35d322134f2a9fa]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.746840 3685 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet c156fd8d4e8f436c97eed32d4b50e1cd. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.748836 3418 tablet_service.cc:1006] Processing CreateTablet for tablet 3337cdf76ae447bd9df561914cede569 (table=TestAuthTokenReacquire-table-7 [id=d6d879c7542741c8a7be7786b888ddd3]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.748960 3418 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 3337cdf76ae447bd9df561914cede569. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.751719 3417 tablet_service.cc:1006] Processing CreateTablet for tablet c07b620efb024973aea4f313eabf4592 (table=TestAuthTokenReacquire-table-6 [id=d70654235532446a85c0dc05f79f3c4f]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.751833 3417 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet c07b620efb024973aea4f313eabf4592. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.752800 3416 tablet_service.cc:1006] Processing CreateTablet for tablet c156fd8d4e8f436c97eed32d4b50e1cd (table=TestAuthTokenReacquire-table-3 [id=a45d2c9dd7fe4a04a35d322134f2a9fa]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.752907 3416 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet c156fd8d4e8f436c97eed32d4b50e1cd. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.763 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.758441 3809 tablet_bootstrap.cc:439] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867: Bootstrap starting. 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.758679 3807 tablet_bootstrap.cc:439] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd: No bootstrap required, opened a new log 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.758849 3807 ts_tablet_manager.cc:1053] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.760128 3809 tablet_bootstrap.cc:590] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867: No blocks or log segments found. Creating new log. 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.761265 3687 tablet_service.cc:1006] Processing CreateTablet for tablet fe1a243ef4c64777b0a73ede1d7f653a (table=TestAuthTokenReacquire-table-4 [id=329758112d6c49beb826456c5e22abca]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.761405 3687 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet fe1a243ef4c64777b0a73ede1d7f653a. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.762100 3810 tablet_bootstrap.cc:439] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9: Bootstrap starting. 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.762431 3245 catalog_manager.cc:1415] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44184: 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "TestAuthTokenReacquire-table-2" 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 11:18:14.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.762773 3245 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAuthTokenReacquire-table-2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.762790 3807 raft_consensus.cc:340] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.762848 3807 raft_consensus.cc:366] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.762879 3807 raft_consensus.cc:705] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7585fc19f9934f8a9df6ecb0d9e664bd, State: Initialized, Role: FOLLOWER 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.763345 3807 consensus_queue.cc:229] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.765395 3810 tablet_bootstrap.cc:590] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9: No blocks or log segments found. Creating new log. 11:18:14.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.766144 3810 log.cc:525] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9: Log is configured to *not* fsync() on all Append() calls 11:18:14.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.767098 3810 tablet_bootstrap.cc:439] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9: No bootstrap required, opened a new log 11:18:14.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.767261 3810 ts_tablet_manager.cc:1053] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s 11:18:14.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.767356 3807 ts_tablet_manager.cc:1081] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent starting tablet: real 0.008s user 0.000s sys 0.005s 11:18:14.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.767571 3807 tablet_bootstrap.cc:439] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd: Bootstrap starting. 11:18:14.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.768445 3807 tablet_bootstrap.cc:590] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd: No blocks or log segments found. Creating new log. 11:18:14.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.768998 3809 log.cc:525] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867: Log is configured to *not* fsync() on all Append() calls 11:18:14.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.770098 3807 tablet_bootstrap.cc:439] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd: No bootstrap required, opened a new log 11:18:14.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.770172 3807 ts_tablet_manager.cc:1053] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s 11:18:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.770587 3809 tablet_bootstrap.cc:439] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867: No bootstrap required, opened a new log 11:18:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.770748 3809 ts_tablet_manager.cc:1053] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s 11:18:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.770905 3807 raft_consensus.cc:340] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.770923 3807 raft_consensus.cc:366] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.770937 3807 raft_consensus.cc:705] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7585fc19f9934f8a9df6ecb0d9e664bd, State: Initialized, Role: FOLLOWER 11:18:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.771114 3807 consensus_queue.cc:229] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.772505 3807 ts_tablet_manager.cc:1081] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s 11:18:14.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.773811 3810 raft_consensus.cc:340] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.773811 3809 raft_consensus.cc:340] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.773875 3810 raft_consensus.cc:366] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.773875 3809 raft_consensus.cc:366] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.773905 3810 raft_consensus.cc:705] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73918a1b139243ff837530f26d7808c9, State: Initialized, Role: FOLLOWER 11:18:14.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.773910 3809 raft_consensus.cc:705] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac72b4ba12248ddad09193985a1e867, State: Initialized, Role: FOLLOWER 11:18:14.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.774166 3807 tablet_bootstrap.cc:439] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd: Bootstrap starting. 11:18:14.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.774348 3809 consensus_queue.cc:229] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.776 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.776340 3807 tablet_bootstrap.cc:590] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd: No blocks or log segments found. Creating new log. 11:18:14.777 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.777199 3810 consensus_queue.cc:229] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.777745 3554 tablet_service.cc:1006] Processing CreateTablet for tablet a5ef36a8cc8249cb98791ad5c9f0ce26 (table=TestAuthTokenReacquire-table-2 [id=7cf5d84d15a24a11abea7c2f5bdd3a06]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.777904 3554 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet a5ef36a8cc8249cb98791ad5c9f0ce26. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.777977 3807 tablet_bootstrap.cc:439] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd: No bootstrap required, opened a new log 11:18:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.778066 3807 ts_tablet_manager.cc:1053] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s 11:18:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.778156 3419 tablet_service.cc:1006] Processing CreateTablet for tablet a5ef36a8cc8249cb98791ad5c9f0ce26 (table=TestAuthTokenReacquire-table-2 [id=7cf5d84d15a24a11abea7c2f5bdd3a06]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.778316 3419 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet a5ef36a8cc8249cb98791ad5c9f0ce26. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.778430 3807 raft_consensus.cc:340] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.778450 3807 raft_consensus.cc:366] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.778462 3807 raft_consensus.cc:705] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7585fc19f9934f8a9df6ecb0d9e664bd, State: Initialized, Role: FOLLOWER 11:18:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.780791 3687 tablet_service.cc:1006] Processing CreateTablet for tablet a5ef36a8cc8249cb98791ad5c9f0ce26 (table=TestAuthTokenReacquire-table-2 [id=7cf5d84d15a24a11abea7c2f5bdd3a06]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.780805 3810 ts_tablet_manager.cc:1081] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s 11:18:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.780934 3687 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet a5ef36a8cc8249cb98791ad5c9f0ce26. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.780994 3810 tablet_bootstrap.cc:439] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9: Bootstrap starting. 11:18:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.781806 3810 tablet_bootstrap.cc:590] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9: No blocks or log segments found. Creating new log. 11:18:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.782377 3807 consensus_queue.cc:229] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.782554 3807 ts_tablet_manager.cc:1081] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s 11:18:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.782677 3807 tablet_bootstrap.cc:439] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd: Bootstrap starting. 11:18:14.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.783430 3807 tablet_bootstrap.cc:590] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd: No blocks or log segments found. Creating new log. 11:18:14.784 [DEBUG - New I/O worker #5] (AsyncKuduClient.java:791) Opened table a45d2c9dd7fe4a04a35d322134f2a9fa 11:18:14.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.783984 3809 ts_tablet_manager.cc:1081] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s 11:18:14.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.784241 3809 tablet_bootstrap.cc:439] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867: Bootstrap starting. 11:18:14.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.784302 3807 tablet_bootstrap.cc:439] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd: No bootstrap required, opened a new log 11:18:14.785 [DEBUG - New I/O worker #5] (AsyncKuduClient.java:791) Opened table d70654235532446a85c0dc05f79f3c4f 11:18:14.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.784353 3807 ts_tablet_manager.cc:1053] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.785081 3810 tablet_bootstrap.cc:439] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9: No bootstrap required, opened a new log 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.785143 3810 ts_tablet_manager.cc:1053] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.785285 3809 tablet_bootstrap.cc:590] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867: No blocks or log segments found. Creating new log. 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.785683 3810 raft_consensus.cc:340] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.785706 3810 raft_consensus.cc:366] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.785719 3810 raft_consensus.cc:705] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73918a1b139243ff837530f26d7808c9, State: Initialized, Role: FOLLOWER 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.785831 3810 consensus_queue.cc:229] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.786145 3807 raft_consensus.cc:340] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.786165 3807 raft_consensus.cc:366] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.786177 3807 raft_consensus.cc:705] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7585fc19f9934f8a9df6ecb0d9e664bd, State: Initialized, Role: FOLLOWER 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.786319 3807 consensus_queue.cc:229] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.786489 3807 ts_tablet_manager.cc:1081] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s 11:18:14.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.786597 3807 tablet_bootstrap.cc:439] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd: Bootstrap starting. 11:18:14.786 [DEBUG - New I/O worker #5] (AsyncKuduClient.java:791) Opened table d6d879c7542741c8a7be7786b888ddd3 11:18:14.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.786996 3809 tablet_bootstrap.cc:439] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867: No bootstrap required, opened a new log 11:18:14.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.787102 3809 ts_tablet_manager.cc:1053] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s 11:18:14.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.787358 3807 tablet_bootstrap.cc:590] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd: No blocks or log segments found. Creating new log. 11:18:14.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.787612 3809 raft_consensus.cc:340] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.787633 3809 raft_consensus.cc:366] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.787647 3809 raft_consensus.cc:705] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac72b4ba12248ddad09193985a1e867, State: Initialized, Role: FOLLOWER 11:18:14.787 [DEBUG - New I/O worker #5] (AsyncKuduClient.java:791) Opened table 329758112d6c49beb826456c5e22abca 11:18:14.790 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.788161 3809 consensus_queue.cc:229] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.792157 3809 ts_tablet_manager.cc:1081] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s 11:18:14.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.792289 3809 tablet_bootstrap.cc:439] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867: Bootstrap starting. 11:18:14.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.793334 3809 tablet_bootstrap.cc:590] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867: No blocks or log segments found. Creating new log. 11:18:14.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.793573 3810 ts_tablet_manager.cc:1081] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s 11:18:14.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.793726 3810 tablet_bootstrap.cc:439] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9: Bootstrap starting. 11:18:14.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.794436 3809 tablet_bootstrap.cc:439] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867: No bootstrap required, opened a new log 11:18:14.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.794600 3810 tablet_bootstrap.cc:590] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9: No blocks or log segments found. Creating new log. 11:18:14.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.794620 3809 ts_tablet_manager.cc:1053] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s 11:18:14.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.795132 3809 raft_consensus.cc:340] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.795271 3809 raft_consensus.cc:366] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.795331 3807 tablet_bootstrap.cc:439] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd: No bootstrap required, opened a new log 11:18:14.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.795389 3807 ts_tablet_manager.cc:1053] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s 11:18:14.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.795470 3809 raft_consensus.cc:705] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac72b4ba12248ddad09193985a1e867, State: Initialized, Role: FOLLOWER 11:18:14.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.795724 3809 consensus_queue.cc:229] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.795830 3807 raft_consensus.cc:340] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.796 [DEBUG - New I/O worker #5] (AsyncKuduClient.java:791) Opened table 7cf5d84d15a24a11abea7c2f5bdd3a06 11:18:14.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.797755 3809 ts_tablet_manager.cc:1081] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s 11:18:14.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.797876 3809 tablet_bootstrap.cc:439] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867: Bootstrap starting. 11:18:14.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.798638 3809 tablet_bootstrap.cc:590] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867: No blocks or log segments found. Creating new log. 11:18:14.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.800065 3807 raft_consensus.cc:366] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.800086 3807 raft_consensus.cc:705] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7585fc19f9934f8a9df6ecb0d9e664bd, State: Initialized, Role: FOLLOWER 11:18:14.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.800457 3809 tablet_bootstrap.cc:439] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867: No bootstrap required, opened a new log 11:18:14.812 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.800509 3809 ts_tablet_manager.cc:1053] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.800719 3807 consensus_queue.cc:229] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.800890 3809 raft_consensus.cc:340] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.800909 3809 raft_consensus.cc:366] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.800922 3809 raft_consensus.cc:705] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac72b4ba12248ddad09193985a1e867, State: Initialized, Role: FOLLOWER 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.801090 3809 consensus_queue.cc:229] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.801580 3809 ts_tablet_manager.cc:1081] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.801707 3809 tablet_bootstrap.cc:439] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867: Bootstrap starting. 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.802454 3809 tablet_bootstrap.cc:590] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867: No blocks or log segments found. Creating new log. 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.803277 3809 tablet_bootstrap.cc:439] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867: No bootstrap required, opened a new log 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.803328 3809 ts_tablet_manager.cc:1053] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.803741 3809 raft_consensus.cc:340] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.805121 3815 raft_consensus.cc:472] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.805238 3815 raft_consensus.cc:494] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.806105 3815 leader_election.cc:296] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105), 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.808274 3815 raft_consensus.cc:472] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.808341 3815 raft_consensus.cc:494] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.808573 3815 leader_election.cc:296] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203), 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.808884 3809 raft_consensus.cc:366] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.808907 3809 raft_consensus.cc:705] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac72b4ba12248ddad09193985a1e867, State: Initialized, Role: FOLLOWER 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.809096 3809 consensus_queue.cc:229] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.810811 3809 ts_tablet_manager.cc:1081] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.810930 3809 tablet_bootstrap.cc:439] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867: Bootstrap starting. 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.811738 3809 tablet_bootstrap.cc:590] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867: No blocks or log segments found. Creating new log. 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.812680 3810 tablet_bootstrap.cc:439] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9: No bootstrap required, opened a new log 11:18:14.813 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.812760 3810 ts_tablet_manager.cc:1053] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.813290 3810 raft_consensus.cc:340] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.813314 3810 raft_consensus.cc:366] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.813330 3810 raft_consensus.cc:705] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73918a1b139243ff837530f26d7808c9, State: Initialized, Role: FOLLOWER 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.814188 3807 ts_tablet_manager.cc:1081] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.814313 3807 tablet_bootstrap.cc:439] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd: Bootstrap starting. 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.814920 3807 tablet_bootstrap.cc:590] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd: No blocks or log segments found. Creating new log. 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.815569 3810 consensus_queue.cc:229] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.816277 3807 tablet_bootstrap.cc:439] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd: No bootstrap required, opened a new log 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.816339 3807 ts_tablet_manager.cc:1053] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.816812 3807 raft_consensus.cc:340] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.816833 3807 raft_consensus.cc:366] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.816846 3807 raft_consensus.cc:705] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7585fc19f9934f8a9df6ecb0d9e664bd, State: Initialized, Role: FOLLOWER 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.816993 3807 consensus_queue.cc:229] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.817080 3809 tablet_bootstrap.cc:439] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867: No bootstrap required, opened a new log 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.817152 3809 ts_tablet_manager.cc:1053] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s 11:18:14.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.817582 3624 tablet.cc:1786] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9: Can't schedule compaction. Clean time has not been advanced past its initial value. 11:18:14.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.820705 3809 raft_consensus.cc:340] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.820732 3809 raft_consensus.cc:366] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.820747 3809 raft_consensus.cc:705] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac72b4ba12248ddad09193985a1e867, State: Initialized, Role: FOLLOWER 11:18:14.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.820896 3809 consensus_queue.cc:229] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.820957 3810 ts_tablet_manager.cc:1081] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s 11:18:14.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.821365 3809 ts_tablet_manager.cc:1081] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s 11:18:14.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.821516 3807 ts_tablet_manager.cc:1081] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s 11:18:14.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.822438 3440 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "1734b9064c4943339961de5d872e236b" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" is_pre_election: true 11:18:14.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.822595 3440 raft_consensus.cc:2346] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 0. 11:18:14.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.822899 3439 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "3337cdf76ae447bd9df561914cede569" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" is_pre_election: true 11:18:14.825 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.822957 3439 raft_consensus.cc:2346] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 0. 11:18:14.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.823302 3638 leader_election.cc:310] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 7585fc19f9934f8a9df6ecb0d9e664bd; no voters: 11:18:14.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.823588 3638 leader_election.cc:310] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 7585fc19f9934f8a9df6ecb0d9e664bd; no voters: 11:18:14.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.823758 3815 raft_consensus.cc:2640] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Leader pre-election won for term 1 11:18:14.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.823799 3815 raft_consensus.cc:472] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 11:18:14.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.823817 3815 raft_consensus.cc:2886] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Advancing to term 1 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.824940 3815 raft_consensus.cc:494] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.825215 3815 leader_election.cc:296] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 election: Requested vote from peers 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105), 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.825278 3815 raft_consensus.cc:2640] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Leader pre-election won for term 1 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.825292 3815 raft_consensus.cc:472] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.825305 3815 raft_consensus.cc:2886] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Advancing to term 1 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.825918 3439 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "1734b9064c4943339961de5d872e236b" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.825964 3439 raft_consensus.cc:2886] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Advancing to term 1 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.826372 3815 raft_consensus.cc:494] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.826912 3440 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "3337cdf76ae447bd9df561914cede569" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.826954 3440 raft_consensus.cc:2886] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Advancing to term 1 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.827628 3439 raft_consensus.cc:2346] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 1. 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.827782 3440 raft_consensus.cc:2346] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 1. 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.827922 3638 leader_election.cc:310] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 7585fc19f9934f8a9df6ecb0d9e664bd; no voters: 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.827980 3815 leader_election.cc:296] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 election: Requested vote from peers 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203), 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.828084 3815 raft_consensus.cc:2640] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Leader election won for term 1 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.828132 3815 raft_consensus.cc:667] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 LEADER]: Becoming Leader. State: Replica: 7585fc19f9934f8a9df6ecb0d9e664bd, State: Running, Role: LEADER 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.828137 3638 leader_election.cc:310] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 7585fc19f9934f8a9df6ecb0d9e664bd; no voters: 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.828419 3815 consensus_queue.cc:206] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.828742 3811 raft_consensus.cc:2640] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Leader election won for term 1 11:18:14.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.828919 3578 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "1734b9064c4943339961de5d872e236b" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73918a1b139243ff837530f26d7808c9" is_pre_election: true 11:18:14.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.828799 3811 raft_consensus.cc:667] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 LEADER]: Becoming Leader. State: Replica: 7585fc19f9934f8a9df6ecb0d9e664bd, State: Running, Role: LEADER 11:18:14.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.829211 3811 consensus_queue.cc:206] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.829404 3577 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "3337cdf76ae447bd9df561914cede569" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73918a1b139243ff837530f26d7808c9" is_pre_election: true 11:18:14.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.829525 3577 raft_consensus.cc:2346] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 0. 11:18:14.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.829666 3576 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "1734b9064c4943339961de5d872e236b" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73918a1b139243ff837530f26d7808c9" 11:18:14.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.829789 3575 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "3337cdf76ae447bd9df561914cede569" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73918a1b139243ff837530f26d7808c9" 11:18:14.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.829824 3575 raft_consensus.cc:2886] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Advancing to term 1 11:18:14.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.830682 3575 raft_consensus.cc:2346] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 1. 11:18:14.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.831066 3639 leader_election.cc:348] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203): Illegal state: must be running to vote when last-logged opid is not known 11:18:14.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.831377 3639 leader_election.cc:348] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203): Illegal state: must be running to vote when last-logged opid is not known 11:18:14.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.831629 3810 tablet_bootstrap.cc:439] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9: Bootstrap starting. 11:18:14.833 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.832727 3810 tablet_bootstrap.cc:590] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9: No blocks or log segments found. Creating new log. 11:18:14.835 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.833488 3245 catalog_manager.cc:3976] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd reported cstate change: term changed from 0 to 1, leader changed from to 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131). New cstate: current_term: 1 leader_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } health_report { overall_health: HEALTHY } } } 11:18:14.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.835764 3810 tablet_bootstrap.cc:439] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9: No bootstrap required, opened a new log 11:18:14.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.836248 3810 ts_tablet_manager.cc:1053] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s 11:18:14.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.836895 3810 raft_consensus.cc:340] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.837116 3810 raft_consensus.cc:366] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.837260 3810 raft_consensus.cc:705] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73918a1b139243ff837530f26d7808c9, State: Initialized, Role: FOLLOWER 11:18:14.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.838114 3810 consensus_queue.cc:229] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.838589 3245 catalog_manager.cc:3976] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd reported cstate change: term changed from 0 to 1, leader changed from to 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131). New cstate: current_term: 1 leader_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } health_report { overall_health: HEALTHY } } } 11:18:14.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.840644 3810 ts_tablet_manager.cc:1081] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s 11:18:14.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.840795 3810 tablet_bootstrap.cc:439] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9: Bootstrap starting. 11:18:14.842 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.841624 3810 tablet_bootstrap.cc:590] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9: No blocks or log segments found. Creating new log. 11:18:14.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.842990 3810 tablet_bootstrap.cc:439] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9: No bootstrap required, opened a new log 11:18:14.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.843228 3810 ts_tablet_manager.cc:1053] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s 11:18:14.843 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.843840 3810 raft_consensus.cc:340] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.843988 3810 raft_consensus.cc:366] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.844136 3810 raft_consensus.cc:705] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73918a1b139243ff837530f26d7808c9, State: Initialized, Role: FOLLOWER 11:18:14.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.844408 3810 consensus_queue.cc:229] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.844729 3810 ts_tablet_manager.cc:1081] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s 11:18:14.844 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.844847 3810 tablet_bootstrap.cc:439] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9: Bootstrap starting. 11:18:14.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.845667 3810 tablet_bootstrap.cc:590] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9: No blocks or log segments found. Creating new log. 11:18:14.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.846729 3810 tablet_bootstrap.cc:439] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9: No bootstrap required, opened a new log 11:18:14.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.846921 3810 ts_tablet_manager.cc:1053] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s 11:18:14.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.847479 3810 raft_consensus.cc:340] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.847647 3810 raft_consensus.cc:366] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:14.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.847764 3810 raft_consensus.cc:705] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73918a1b139243ff837530f26d7808c9, State: Initialized, Role: FOLLOWER 11:18:14.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.847998 3810 consensus_queue.cc:229] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.851 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.849400 3810 ts_tablet_manager.cc:1081] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s 11:18:14.854 [DEBUG - New I/O worker #4] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:14.852 [DEBUG - New I/O worker #3] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:14.857 [DEBUG - New I/O worker #5] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:14.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.868891 3815 raft_consensus.cc:472] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 11:18:14.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.869061 3815 raft_consensus.cc:494] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.869453 3815 leader_election.cc:296] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105), 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) 11:18:14.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.870146 3440 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "c07b620efb024973aea4f313eabf4592" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" is_pre_election: true 11:18:14.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.870159 3575 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "c07b620efb024973aea4f313eabf4592" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73918a1b139243ff837530f26d7808c9" is_pre_election: true 11:18:14.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.870252 3440 raft_consensus.cc:2346] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 0. 11:18:14.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.870259 3575 raft_consensus.cc:2346] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 0. 11:18:14.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.870637 3638 leader_election.cc:310] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 7585fc19f9934f8a9df6ecb0d9e664bd; no voters: 11:18:14.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.870849 3815 raft_consensus.cc:2640] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Leader pre-election won for term 1 11:18:14.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.870867 3815 raft_consensus.cc:472] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 11:18:14.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.870879 3815 raft_consensus.cc:2886] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Advancing to term 1 11:18:14.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.871814 3815 raft_consensus.cc:494] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.872130 3815 leader_election.cc:296] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 election: Requested vote from peers 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105), 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) 11:18:14.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.872444 3440 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "c07b620efb024973aea4f313eabf4592" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" 11:18:14.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.872473 3575 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "c07b620efb024973aea4f313eabf4592" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73918a1b139243ff837530f26d7808c9" 11:18:14.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.872511 3575 raft_consensus.cc:2886] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Advancing to term 1 11:18:14.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.872496 3440 raft_consensus.cc:2886] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Advancing to term 1 11:18:14.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.873404 3575 raft_consensus.cc:2346] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 1. 11:18:14.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.873405 3440 raft_consensus.cc:2346] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 1. 11:18:14.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.873673 3638 leader_election.cc:310] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 7585fc19f9934f8a9df6ecb0d9e664bd; no voters: 11:18:14.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.873877 3815 raft_consensus.cc:2640] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Leader election won for term 1 11:18:14.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.873931 3815 raft_consensus.cc:667] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 LEADER]: Becoming Leader. State: Replica: 7585fc19f9934f8a9df6ecb0d9e664bd, State: Running, Role: LEADER 11:18:14.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.874166 3815 consensus_queue.cc:206] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.874 [DEBUG - New I/O worker #6] (Negotiator.java:764) Authenticated connection [id: 0xd3f4f07e, /127.0.0.1:44192 => /127.3.15.189:41941] using TOKEN/null 11:18:14.880 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.875808 3240 catalog_manager.cc:3976] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd reported cstate change: term changed from 0 to 1, leader changed from to 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131). New cstate: current_term: 1 leader_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } health_report { overall_health: HEALTHY } } } 11:18:14.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.881772 3818 raft_consensus.cc:472] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 11:18:14.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.881877 3818 raft_consensus.cc:494] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.882918 3818 leader_election.cc:296] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105), 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) 11:18:14.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.887660 3440 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "fe1a243ef4c64777b0a73ede1d7f653a" candidate_uuid: "73918a1b139243ff837530f26d7808c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" is_pre_election: true 11:18:14.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.887743 3440 raft_consensus.cc:2346] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73918a1b139243ff837530f26d7808c9 in term 0. 11:18:14.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.888125 3509 leader_election.cc:310] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 73918a1b139243ff837530f26d7808c9; no voters: 11:18:14.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.888444 3486 tablet.cc:1786] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867: Can't schedule compaction. Clean time has not been advanced past its initial value. 11:18:14.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.888638 3707 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "fe1a243ef4c64777b0a73ede1d7f653a" candidate_uuid: "73918a1b139243ff837530f26d7808c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" is_pre_election: true 11:18:14.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.888738 3707 raft_consensus.cc:2346] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73918a1b139243ff837530f26d7808c9 in term 0. 11:18:14.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.889077 3818 raft_consensus.cc:2640] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Leader pre-election won for term 1 11:18:14.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.889117 3818 raft_consensus.cc:472] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 11:18:14.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.889135 3818 raft_consensus.cc:2886] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Advancing to term 1 11:18:14.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.890080 3818 raft_consensus.cc:494] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.890319 3818 leader_election.cc:296] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [CANDIDATE]: Term 1 election: Requested vote from peers 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105), 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) 11:18:14.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.890653 3440 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "fe1a243ef4c64777b0a73ede1d7f653a" candidate_uuid: "73918a1b139243ff837530f26d7808c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" 11:18:14.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.890687 3707 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "fe1a243ef4c64777b0a73ede1d7f653a" candidate_uuid: "73918a1b139243ff837530f26d7808c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" 11:18:14.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.890718 3440 raft_consensus.cc:2886] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Advancing to term 1 11:18:14.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.890730 3707 raft_consensus.cc:2886] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Advancing to term 1 11:18:14.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.891461 3440 raft_consensus.cc:2346] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73918a1b139243ff837530f26d7808c9 in term 1. 11:18:14.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.891626 3707 raft_consensus.cc:2346] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73918a1b139243ff837530f26d7808c9 in term 1. 11:18:14.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.891747 3509 leader_election.cc:310] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 73918a1b139243ff837530f26d7808c9; no voters: 11:18:14.891 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.891882 3818 raft_consensus.cc:2640] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Leader election won for term 1 11:18:14.892 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.892078 3818 raft_consensus.cc:667] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 1 LEADER]: Becoming Leader. State: Replica: 73918a1b139243ff837530f26d7808c9, State: Running, Role: LEADER 11:18:14.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.892385 3818 consensus_queue.cc:206] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.895489 3240 catalog_manager.cc:3976] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 reported cstate change: term changed from 0 to 1, leader changed from to 73918a1b139243ff837530f26d7808c9 (127.3.15.130). New cstate: current_term: 1 leader_uuid: "73918a1b139243ff837530f26d7808c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } health_report { overall_health: UNKNOWN } } } 11:18:14.908 [DEBUG - New I/O worker #6] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer 11:18:14.908 [INFO - New I/O worker #6] (AsyncKuduClient.java:2058) Invalidating location master-127.3.15.189:41941(127.3.15.189:41941) for tablet Kudu Master: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer 11:18:14.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.908999 3228 connection.cc:647] server connection from 127.0.0.1:44192 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44192): Connection reset by peer (error 104) 11:18:14.923 [DEBUG - New I/O worker #8] (Negotiator.java:764) Authenticated connection [id: 0x213be60e, /127.0.0.1:37746 => /127.3.15.188:39851] using TOKEN/null 11:18:14.924 [DEBUG - New I/O worker #7] (Negotiator.java:291) peer /127.3.15.189:41941 sent connection negotiation error: Not authorized: invalid token signature 11:18:14.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.923195 3780 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44194: invalid token signature 11:18:14.928 [DEBUG - New I/O worker #1] (Negotiator.java:764) Authenticated connection [id: 0x3b495040, /127.0.0.1:56226 => /127.3.15.190:34417] using TOKEN/null 11:18:14.928 [DEBUG - New I/O worker #7] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:14.940 [DEBUG - New I/O worker #3] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.188 11:18:14.941 [DEBUG - New I/O worker #4] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.190 11:18:14.952 [DEBUG - New I/O worker #2] (Negotiator.java:764) Authenticated connection [id: 0xc3e50a7d, /127.0.0.1:44200 => /127.3.15.189:41941] using TOKEN/null 11:18:14.953 [DEBUG - New I/O worker #3] (Negotiator.java:764) Authenticated connection [id: 0xdd016788, /127.0.0.1:37752 => /127.3.15.188:39851] using SASL/GSSAPI 11:18:14.954 [DEBUG - New I/O worker #4] (Negotiator.java:764) Authenticated connection [id: 0xa1c12473, /127.0.0.1:56232 => /127.3.15.190:34417] using SASL/GSSAPI 11:18:14.957 [DEBUG - New I/O worker #5] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.189 11:18:14.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.950693 3818 raft_consensus.cc:472] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 11:18:14.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.950836 3818 raft_consensus.cc:494] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.951333 3818 leader_election.cc:296] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105), 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.951910 3707 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "a5ef36a8cc8249cb98791ad5c9f0ce26" candidate_uuid: "73918a1b139243ff837530f26d7808c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" is_pre_election: true 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.952011 3707 raft_consensus.cc:2346] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73918a1b139243ff837530f26d7808c9 in term 0. 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.952728 3440 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "a5ef36a8cc8249cb98791ad5c9f0ce26" candidate_uuid: "73918a1b139243ff837530f26d7808c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" is_pre_election: true 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.952826 3440 raft_consensus.cc:2346] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73918a1b139243ff837530f26d7808c9 in term 0. 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.953253 3510 leader_election.cc:310] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73918a1b139243ff837530f26d7808c9, 7585fc19f9934f8a9df6ecb0d9e664bd; no voters: 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.953704 3818 raft_consensus.cc:2640] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Leader pre-election won for term 1 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.953727 3818 raft_consensus.cc:472] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.953743 3818 raft_consensus.cc:2886] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Advancing to term 1 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.954847 3818 raft_consensus.cc:494] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.955168 3818 leader_election.cc:296] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [CANDIDATE]: Term 1 election: Requested vote from peers 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105), 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.955569 3707 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "a5ef36a8cc8249cb98791ad5c9f0ce26" candidate_uuid: "73918a1b139243ff837530f26d7808c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.955631 3707 raft_consensus.cc:2886] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Advancing to term 1 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.956496 3707 raft_consensus.cc:2346] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73918a1b139243ff837530f26d7808c9 in term 1. 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.957080 3440 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "a5ef36a8cc8249cb98791ad5c9f0ce26" candidate_uuid: "73918a1b139243ff837530f26d7808c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.957144 3440 raft_consensus.cc:2886] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Advancing to term 1 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.958009 3440 raft_consensus.cc:2346] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73918a1b139243ff837530f26d7808c9 in term 1. 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.958433 3510 leader_election.cc:310] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73918a1b139243ff837530f26d7808c9, 7585fc19f9934f8a9df6ecb0d9e664bd; no voters: 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.958711 3818 raft_consensus.cc:2640] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Leader election won for term 1 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.958777 3818 raft_consensus.cc:667] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 1 LEADER]: Becoming Leader. State: Replica: 73918a1b139243ff837530f26d7808c9, State: Running, Role: LEADER 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.958951 3818 consensus_queue.cc:206] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.961941 3245 catalog_manager.cc:3976] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 reported cstate change: term changed from 0 to 1, leader changed from to 73918a1b139243ff837530f26d7808c9 (127.3.15.130). New cstate: current_term: 1 leader_uuid: "73918a1b139243ff837530f26d7808c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } health_report { overall_health: UNKNOWN } } } 11:18:14.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.966732 3753 tablet.cc:1786] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd: Can't schedule compaction. Clean time has not been advanced past its initial value. 11:18:14.971 [DEBUG - New I/O worker #2] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:14.972 [DEBUG - New I/O worker #2] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=IsCreateTableDone, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=44), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [8ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [10ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [36ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [41ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=null) after lookup 11:18:14.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.969790 3816 raft_consensus.cc:472] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 11:18:14.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.969903 3816 raft_consensus.cc:494] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.970919 3816 leader_election.cc:296] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203), 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) 11:18:14.982 [DEBUG - New I/O worker #5] (Negotiator.java:764) Authenticated connection [id: 0x4c1ab8a1, /127.0.0.1:44206 => /127.3.15.189:41941] using SASL/GSSAPI 11:18:14.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.978545 3575 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "c156fd8d4e8f436c97eed32d4b50e1cd" candidate_uuid: "3ac72b4ba12248ddad09193985a1e867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73918a1b139243ff837530f26d7808c9" is_pre_election: true 11:18:14.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.978632 3575 raft_consensus.cc:2346] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ac72b4ba12248ddad09193985a1e867 in term 0. 11:18:14.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.978996 3372 leader_election.cc:310] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 73918a1b139243ff837530f26d7808c9; no voters: 11:18:14.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.979326 3816 raft_consensus.cc:2640] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Leader pre-election won for term 1 11:18:14.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.979351 3816 raft_consensus.cc:472] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 11:18:14.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.979365 3816 raft_consensus.cc:2886] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Advancing to term 1 11:18:14.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.979790 3707 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "c156fd8d4e8f436c97eed32d4b50e1cd" candidate_uuid: "3ac72b4ba12248ddad09193985a1e867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" is_pre_election: true 11:18:14.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.979862 3707 raft_consensus.cc:2346] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ac72b4ba12248ddad09193985a1e867 in term 0. 11:18:14.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.980423 3816 raft_consensus.cc:494] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.980729 3816 leader_election.cc:296] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [CANDIDATE]: Term 1 election: Requested vote from peers 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203), 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) 11:18:14.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.981323 3707 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "c156fd8d4e8f436c97eed32d4b50e1cd" candidate_uuid: "3ac72b4ba12248ddad09193985a1e867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" 11:18:14.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.981369 3707 raft_consensus.cc:2886] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Advancing to term 1 11:18:14.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.982193 3707 raft_consensus.cc:2346] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac72b4ba12248ddad09193985a1e867 in term 1. 11:18:14.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.982416 3575 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "c156fd8d4e8f436c97eed32d4b50e1cd" candidate_uuid: "3ac72b4ba12248ddad09193985a1e867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73918a1b139243ff837530f26d7808c9" 11:18:14.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.982467 3575 raft_consensus.cc:2886] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Advancing to term 1 11:18:14.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.983371 3575 raft_consensus.cc:2346] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac72b4ba12248ddad09193985a1e867 in term 1. 11:18:14.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.983712 3372 leader_election.cc:310] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 73918a1b139243ff837530f26d7808c9; no voters: 11:18:14.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.983916 3816 raft_consensus.cc:2640] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Leader election won for term 1 11:18:14.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.984140 3816 raft_consensus.cc:667] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 1 LEADER]: Becoming Leader. State: Replica: 3ac72b4ba12248ddad09193985a1e867, State: Running, Role: LEADER 11:18:14.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.984462 3816 consensus_queue.cc:206] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:14.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.987092 3240 catalog_manager.cc:3976] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 reported cstate change: term changed from 0 to 1, leader changed from to 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129). New cstate: current_term: 1 leader_uuid: "3ac72b4ba12248ddad09193985a1e867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } health_report { overall_health: UNKNOWN } } } 11:18:14.993 [INFO - New I/O worker #5] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:14.999 [DEBUG - New I/O worker #5] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:15.000 [DEBUG - New I/O worker #5] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=IsCreateTableDone, tablet=Kudu Master, attempt=2, TimeoutTracker(timeout=50000, elapsed=94), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [2ms] received response from server master-127.3.15.189:41941: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [2ms] delaying RPC due to: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [29ms] refreshing cache from master, [29ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [29ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [58ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [62ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [66ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [91ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=Deferred@1155822847(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduClient$15@3c1c4e9e, errback=org.apache.kudu.client.AsyncKuduClient$12@78f6904f)) after lookup 11:18:15.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:14.994675 3240 catalog_manager.cc:1415] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44206: 11:18:15.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "TestAuthTokenReacquire-table-5" 11:18:15.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 11:18:15.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:15.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:15.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 11:18:15.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:15.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:15.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.009 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:15.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:15.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 11:18:15.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 11:18:15.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:15.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:14.994974 3240 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAuthTokenReacquire-table-5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 11:18:15.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.002729 3419 tablet_service.cc:1006] Processing CreateTablet for tablet 8a123165cc324cbe9ea0249a3891a94b (table=TestAuthTokenReacquire-table-5 [id=3b699b77de1f4f1a8674230cc14f7631]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:15.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.002760 3687 tablet_service.cc:1006] Processing CreateTablet for tablet 8a123165cc324cbe9ea0249a3891a94b (table=TestAuthTokenReacquire-table-5 [id=3b699b77de1f4f1a8674230cc14f7631]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:15.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.002777 3554 tablet_service.cc:1006] Processing CreateTablet for tablet 8a123165cc324cbe9ea0249a3891a94b (table=TestAuthTokenReacquire-table-5 [id=3b699b77de1f4f1a8674230cc14f7631]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:15.020 [DEBUG - New I/O worker #5] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:15.021 [DEBUG - New I/O worker #5] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=111), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [28ms] Sub RPC ConnectToMaster: waiting for new authn token, [28ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [54ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [86ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [110ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=null) after lookup 11:18:15.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.002940 3687 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 8a123165cc324cbe9ea0249a3891a94b. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:15.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.002944 3419 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 8a123165cc324cbe9ea0249a3891a94b. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:15.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.002946 3554 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 8a123165cc324cbe9ea0249a3891a94b. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:15.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.004914 3807 tablet_bootstrap.cc:439] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd: Bootstrap starting. 11:18:15.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.004992 3810 tablet_bootstrap.cc:439] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9: Bootstrap starting. 11:18:15.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.005089 3809 tablet_bootstrap.cc:439] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867: Bootstrap starting. 11:18:15.021 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.005694 3807 tablet_bootstrap.cc:590] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd: No blocks or log segments found. Creating new log. 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.005867 3809 tablet_bootstrap.cc:590] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867: No blocks or log segments found. Creating new log. 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.005867 3810 tablet_bootstrap.cc:590] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9: No blocks or log segments found. Creating new log. 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.008301 3810 tablet_bootstrap.cc:439] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9: No bootstrap required, opened a new log 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.008394 3810 ts_tablet_manager.cc:1053] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.009124 3810 raft_consensus.cc:340] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.009169 3810 raft_consensus.cc:366] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.009186 3810 raft_consensus.cc:705] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73918a1b139243ff837530f26d7808c9, State: Initialized, Role: FOLLOWER 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.009446 3810 consensus_queue.cc:229] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.011080 3810 ts_tablet_manager.cc:1081] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.011929 3807 tablet_bootstrap.cc:439] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd: No bootstrap required, opened a new log 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.012245 3807 ts_tablet_manager.cc:1053] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.012552 3809 tablet_bootstrap.cc:439] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867: No bootstrap required, opened a new log 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.012645 3809 ts_tablet_manager.cc:1053] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.012820 3807 raft_consensus.cc:340] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.012840 3807 raft_consensus.cc:366] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.012853 3807 raft_consensus.cc:705] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7585fc19f9934f8a9df6ecb0d9e664bd, State: Initialized, Role: FOLLOWER 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.012979 3807 consensus_queue.cc:229] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.013226 3807 ts_tablet_manager.cc:1081] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.014748 3809 raft_consensus.cc:340] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.014780 3809 raft_consensus.cc:366] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.014794 3809 raft_consensus.cc:705] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac72b4ba12248ddad09193985a1e867, State: Initialized, Role: FOLLOWER 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.014915 3809 consensus_queue.cc:229] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.015169 3809 ts_tablet_manager.cc:1081] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.022223 3241 catalog_manager.cc:1415] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44206: 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "TestAuthTokenReacquire-table-0" 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 11:18:15.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:15.023 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:15.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 11:18:15.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 11:18:15.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:15.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.027 [DEBUG - New I/O worker #8] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:15.032 [DEBUG - New I/O worker #4] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:15.032 [DEBUG - New I/O worker #2] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:15.029 [DEBUG - New I/O worker #1] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:15.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:15.022559 3241 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAuthTokenReacquire-table-0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.032058 3687 tablet_service.cc:1006] Processing CreateTablet for tablet 3abd1116f6db4d80b285597f66adbb17 (table=TestAuthTokenReacquire-table-0 [id=0ccfb0ed3bc24bcf8b70085648f640fc]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.032191 3554 tablet_service.cc:1006] Processing CreateTablet for tablet 3abd1116f6db4d80b285597f66adbb17 (table=TestAuthTokenReacquire-table-0 [id=0ccfb0ed3bc24bcf8b70085648f640fc]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.032270 3687 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 3abd1116f6db4d80b285597f66adbb17. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.032397 3554 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 3abd1116f6db4d80b285597f66adbb17. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.033334 3419 tablet_service.cc:1006] Processing CreateTablet for tablet 3abd1116f6db4d80b285597f66adbb17 (table=TestAuthTokenReacquire-table-0 [id=0ccfb0ed3bc24bcf8b70085648f640fc]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.033536 3419 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 3abd1116f6db4d80b285597f66adbb17. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.034505 3810 tablet_bootstrap.cc:439] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9: Bootstrap starting. 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.034534 3807 tablet_bootstrap.cc:439] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd: Bootstrap starting. 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.035341 3810 tablet_bootstrap.cc:590] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9: No blocks or log segments found. Creating new log. 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.035372 3807 tablet_bootstrap.cc:590] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd: No blocks or log segments found. Creating new log. 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.035568 3809 tablet_bootstrap.cc:439] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867: Bootstrap starting. 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.036514 3807 tablet_bootstrap.cc:439] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd: No bootstrap required, opened a new log 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.036532 3809 tablet_bootstrap.cc:590] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867: No blocks or log segments found. Creating new log. 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.036571 3807 ts_tablet_manager.cc:1053] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.036741 3810 tablet_bootstrap.cc:439] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9: No bootstrap required, opened a new log 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.036813 3810 ts_tablet_manager.cc:1053] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.037075 3807 raft_consensus.cc:340] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.037096 3807 raft_consensus.cc:366] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.037109 3807 raft_consensus.cc:705] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7585fc19f9934f8a9df6ecb0d9e664bd, State: Initialized, Role: FOLLOWER 11:18:15.029 [DEBUG - New I/O worker #3] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:15.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.037226 3807 consensus_queue.cc:229] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.037317 3810 raft_consensus.cc:340] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.037339 3810 raft_consensus.cc:366] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:15.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.037353 3810 raft_consensus.cc:705] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73918a1b139243ff837530f26d7808c9, State: Initialized, Role: FOLLOWER 11:18:15.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.037441 3807 ts_tablet_manager.cc:1081] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s 11:18:15.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.037475 3810 consensus_queue.cc:229] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.037674 3810 ts_tablet_manager.cc:1081] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s 11:18:15.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.037698 3809 tablet_bootstrap.cc:439] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867: No bootstrap required, opened a new log 11:18:15.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.037765 3809 ts_tablet_manager.cc:1053] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s 11:18:15.043 [DEBUG - New I/O worker #5] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:15.044 [INFO - New I/O worker #5] (AsyncKuduClient.java:2058) Invalidating location master-127.3.15.189:41941(127.3.15.189:41941) for tablet Kudu Master: connection disconnected 11:18:15.044 [INFO - New I/O worker #5] (AsyncKuduClient.java:2058) Invalidating location master-127.3.15.189:41941(127.3.15.189:41941) for tablet Kudu Master: connection disconnected 11:18:15.044 [DEBUG - New I/O worker #5] (RemoteTablet.java:131) tablet Kudu Master already removed ts master-127.3.15.189:41941, size left is 0 11:18:15.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.038288 3809 raft_consensus.cc:340] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.038311 3809 raft_consensus.cc:366] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:15.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.038326 3809 raft_consensus.cc:705] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac72b4ba12248ddad09193985a1e867, State: Initialized, Role: FOLLOWER 11:18:15.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.038455 3809 consensus_queue.cc:229] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.038681 3809 ts_tablet_manager.cc:1081] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s 11:18:15.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:15.043187 3228 connection.cc:320] Error closing socket: Network error: TlsSocket::Close: Connection reset by peer 11:18:15.075 [DEBUG - New I/O worker #8] (Negotiator.java:764) Authenticated connection [id: 0x8a4d61a9, /127.0.0.1:44212 => /127.3.15.189:41941] using TOKEN/null 11:18:15.077 [DEBUG - New I/O worker #6] (Negotiator.java:764) Authenticated connection [id: 0x69edf77b, /127.0.0.1:37758 => /127.3.15.188:39851] using TOKEN/null 11:18:15.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:15.074913 3782 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56238: invalid token signature 11:18:15.075 [DEBUG - New I/O worker #7] (Negotiator.java:291) peer /127.3.15.190:34417 sent connection negotiation error: Not authorized: invalid token signature 11:18:15.083 [DEBUG - New I/O worker #7] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:15.085 [DEBUG - New I/O worker #8] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:15.085 [DEBUG - New I/O worker #8] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=IsCreateTableDone, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=40), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [35ms] Sub RPC ConnectToMaster: waiting for new authn token, [36ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [36ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=null) after lookup 11:18:15.089 [DEBUG - New I/O worker #8] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:15.089 [DEBUG - New I/O worker #1] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.188 11:18:15.091 [DEBUG - New I/O worker #3] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.189 11:18:15.090 [DEBUG - New I/O worker #2] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.190 11:18:15.089 [DEBUG - New I/O worker #8] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=CreateTable, tablet=Kudu Master, attempt=3, TimeoutTracker(timeout=50000, elapsed=180), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [28ms] Sub RPC ConnectToMaster: waiting for new authn token, [28ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [54ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [86ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [110ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [112ms] sending RPC to server master-127.3.15.189:41941, [135ms] received response from server master-127.3.15.189:41941: Network error: connection disconnected, [135ms] delaying RPC due to: Network error: connection disconnected, [159ms] refreshing cache from master, [160ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [160ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [160ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [175ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [178ms] Sub RPC ConnectToMaster: waiting for new authn token, [179ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=Deferred@439169025(state=PENDING, result=null, callback=(continuation of Deferred@161440493 after retry RPC@1416920555), errback=(continuation of Deferred@161440493 after retry RPC@1416920555))) after lookup 11:18:15.094 [DEBUG - New I/O worker #8] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:15.094 [DEBUG - New I/O worker #8] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=Kudu Master, attempt=2, TimeoutTracker(timeout=50000, elapsed=71), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [21ms] received response from server master-127.3.15.189:41941: Network error: connection disconnected, [21ms] delaying RPC due to: Network error: connection disconnected, [46ms] refreshing cache from master, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [62ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [64ms] Sub RPC ConnectToMaster: waiting for new authn token, [71ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=Deferred@1832732163(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduClient$6@1e63ce9d -> (continuation of Deferred@1821986742 after org.apache.kudu.client.AsyncKuduClient$1@1b75fceb@460717291), errback=passthrough -> (continuation of Deferred@1821986742 after org.apache.kudu.client.AsyncKuduClient$1@1b75fceb@460717291))) after lookup 11:18:15.105 [DEBUG - New I/O worker #6] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:15.107 [DEBUG - New I/O worker #8] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:15.108 [DEBUG - New I/O worker #1] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state NEGOTIATING due to: connection disconnected 11:18:15.109 [INFO - New I/O worker #1] (ConnectToCluster.java:361) Unable to connect to master 127.3.15.188:39851: connection disconnected 11:18:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:15.109457 3781 negotiation.cc:313] Failed RPC negotiation. Trace: 11:18:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.086971 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:37764 11:18:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.087196 (+ 225us) server_negotiation.cc:184] Beginning negotiation 11:18:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.087200 (+ 4us) server_negotiation.cc:373] Waiting for connection header 11:18:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.087213 (+ 13us) server_negotiation.cc:381] Connection header received 11:18:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.088888 (+ 1675us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 11:18:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.088893 (+ 5us) server_negotiation.cc:420] Received NEGOTIATE request from client 11:18:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.088969 (+ 76us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 11:18:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.089072 (+ 103us) server_negotiation.cc:205] Negotiated authn=SASL 11:18:15.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.097095 (+ 8023us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 11:18:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.097252 (+ 157us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 11:18:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.099942 (+ 2690us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 11:18:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.100547 (+ 605us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 11:18:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.100670 (+ 123us) server_negotiation.cc:589] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 11:18:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.108635 (+ 7965us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37764: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:37764): Cannot send after transport endpoint shutdown (error 108) 11:18:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":95} 11:18:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:15.110723 3782 negotiation.cc:313] Failed RPC negotiation. Trace: 11:18:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.087382 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:56244 11:18:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.087571 (+ 189us) server_negotiation.cc:184] Beginning negotiation 11:18:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.087575 (+ 4us) server_negotiation.cc:373] Waiting for connection header 11:18:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.087590 (+ 15us) server_negotiation.cc:381] Connection header received 11:18:15.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.089205 (+ 1615us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.089211 (+ 6us) server_negotiation.cc:420] Received NEGOTIATE request from client 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.089276 (+ 65us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.089341 (+ 65us) server_negotiation.cc:205] Negotiated authn=SASL 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.093980 (+ 4639us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.094185 (+ 205us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.097363 (+ 3178us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.097917 (+ 554us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.097981 (+ 64us) server_negotiation.cc:589] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.101764 (+ 3783us) server_negotiation.cc:337] Received SASL_INITIATE NegotiatePB request 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.101794 (+ 30us) server_negotiation.cc:746] Received SASL_INITIATE request from client 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.101797 (+ 3us) server_negotiation.cc:757] Client requested to use mechanism: GSSAPI 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.101843 (+ 46us) server_negotiation.cc:786] Calling sasl_server_start() 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.103223 (+ 1380us) server_negotiation.cc:349] Sending SASL_CHALLENGE NegotiatePB response 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.110286 (+ 7063us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56244: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:56244): Cannot send after transport endpoint shutdown (error 108) 11:18:15.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":62} 11:18:15.112 [DEBUG - New I/O worker #2] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state NEGOTIATING due to: connection disconnected 11:18:15.113 [INFO - New I/O worker #2] (ConnectToCluster.java:361) Unable to connect to master 127.3.15.190:34417: connection disconnected 11:18:15.115 [DEBUG - New I/O worker #3] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state NEGOTIATING due to: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer 11:18:15.115 [INFO - New I/O worker #3] (ConnectToCluster.java:361) Unable to connect to master 127.3.15.189:41941: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer 11:18:15.115 [WARN - New I/O worker #3] (ConnectToCluster.java:281) Unable to find the leader master 127.3.15.188:39851,127.3.15.190:34417,127.3.15.189:41941; will retry 11:18:15.118 [DEBUG - New I/O worker #4] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.188 11:18:15.118 [INFO - New I/O worker #3] (AsyncKuduClient.java:2058) Invalidating location master-127.3.15.189:41941(127.3.15.189:41941) for tablet Kudu Master: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer 11:18:15.123 [DEBUG - New I/O worker #6] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.189 11:18:15.129 [DEBUG - New I/O worker #5] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.190 11:18:15.134 [DEBUG - New I/O worker #6] (Negotiator.java:764) Authenticated connection [id: 0x531b928a, /127.0.0.1:44224 => /127.3.15.189:41941] using SASL/GSSAPI 11:18:15.136 [INFO - New I/O worker #3] (AsyncKuduClient.java:2058) Invalidating location master-127.3.15.189:41941(127.3.15.189:41941) for tablet Kudu Master: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer 11:18:15.137 [DEBUG - New I/O worker #4] (Negotiator.java:764) Authenticated connection [id: 0xcc3309b5, /127.0.0.1:37770 => /127.3.15.188:39851] using SASL/GSSAPI 11:18:15.138 [DEBUG - New I/O worker #5] (Negotiator.java:764) Authenticated connection [id: 0x4566004c, /127.0.0.1:56250 => /127.3.15.190:34417] using SASL/GSSAPI 11:18:15.140 [INFO - New I/O worker #6] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:15.144 [DEBUG - New I/O worker #6] (ConnectToCluster.java:331) Callback already invoked, discarding response(role: LEADER ca_cert_der: "0\202\001\3170\202\0018\240\003\002\001\002\002\t\000\256\311\337\365z\322\26540\r\006\t*\206H\206\367\r\001\001\v\005\0000\0271\0250\023\006\003U\004\003\023\fkudu-ipki-ca0\036\027\r190406111809Z\027\r290403111809Z0\0271\0250\023\006\003U\004\003\023\fkudu-ipki-ca0\201\2370\r\006\t*\206H\206\367\r\001\001\001\005\000\003\201\215\0000\201\211\002\201\201\000\275<\341\223\304\316\333D\353\323\334\004\346;\372\323i\3144\206\024\214dP\2272/4\327k\214\362)\004\363\254u\001\235\230\tA\005\224\367\222\035\003\213\244\371\241\031T4\264\321\325\376*\327\217_\3571{\221\301\223\036J\341m=\3363\251q\21535r\030\253Vh}9:\256\200\246\225\212\220\230\366\302X\317\301\036\202Ey0\341\334\232b\252\355\231{T\316\317\2475\237\267W\f\344o\216\336\261\002\003\001\000\001\243#0!0\016\006\003U\035\017\001\001\377\004\004\003\002\002\0040\017\006\003U\035\023\001\001\377\004\0050\003\001\001\3770\r\006\t*\206H\206\367\r\001\001\v\005\000\003\201\201\000\213\206\305\253\355,\2264\r1\353\002KN\376\341\216Rk\310\324U_\247\325\305\2067\3016J\313\366\376\336\006\246\376\274wr\323\316>!\003\021+\324xl\316OG#\022\017\364L\314\361<\037U\364\353\333\ff\216\001\222\353\300C\326\265\275\274\224\256\246~+H\211+KH|\347\270J\374\342\247\245\024\335\357\233V\203N\230\362\016\031\310\342\277\270\340S\262\234R[}\376\340R\356V\366@\304(" authn_token { token_data: "\b\370\225\242\345\005\032\f\n\ntest-admin" signature: "C\021\312\2200\257-XIB\021\271;n\342\304sm5\322?\241C\277W\200\217\304\301X\260\037p\021\355\034\003aG?\337\tR\2176Tg\003J\r\202\212\203\261\034\300\252\321\273\233\321\221\334\032" signing_key_seq_num: 0 } master_addrs { host: "127.3.15.190" port: 34417 } master_addrs { host: "127.3.15.189" port: 41941 } master_addrs { host: "127.3.15.188" port: 39851 } ) from 127.3.15.190:34417 11:18:15.146 [DEBUG - New I/O worker #6] (ConnectToCluster.java:331) Callback already invoked, discarding response(role: LEADER ca_cert_der: "0\202\001\3170\202\0018\240\003\002\001\002\002\t\000\256\311\337\365z\322\26540\r\006\t*\206H\206\367\r\001\001\v\005\0000\0271\0250\023\006\003U\004\003\023\fkudu-ipki-ca0\036\027\r190406111809Z\027\r290403111809Z0\0271\0250\023\006\003U\004\003\023\fkudu-ipki-ca0\201\2370\r\006\t*\206H\206\367\r\001\001\001\005\000\003\201\215\0000\201\211\002\201\201\000\275<\341\223\304\316\333D\353\323\334\004\346;\372\323i\3144\206\024\214dP\2272/4\327k\214\362)\004\363\254u\001\235\230\tA\005\224\367\222\035\003\213\244\371\241\031T4\264\321\325\376*\327\217_\3571{\221\301\223\036J\341m=\3363\251q\21535r\030\253Vh}9:\256\200\246\225\212\220\230\366\302X\317\301\036\202Ey0\341\334\232b\252\355\231{T\316\317\2475\237\267W\f\344o\216\336\261\002\003\001\000\001\243#0!0\016\006\003U\035\017\001\001\377\004\004\003\002\002\0040\017\006\003U\035\023\001\001\377\004\0050\003\001\001\3770\r\006\t*\206H\206\367\r\001\001\v\005\000\003\201\201\000\213\206\305\253\355,\2264\r1\353\002KN\376\341\216Rk\310\324U_\247\325\305\2067\3016J\313\366\376\336\006\246\376\274wr\323\316>!\003\021+\324xl\316OG#\022\017\364L\314\361<\037U\364\353\333\ff\216\001\222\353\300C\326\265\275\274\224\256\246~+H\211+KH|\347\270J\374\342\247\245\024\335\357\233V\203N\230\362\016\031\310\342\277\270\340S\262\234R[}\376\340R\356V\366@\304(" authn_token { token_data: "\b\370\225\242\345\005\032\f\n\ntest-admin" signature: "C\021\312\2200\257-XIB\021\271;n\342\304sm5\322?\241C\277W\200\217\304\301X\260\037p\021\355\034\003aG?\337\tR\2176Tg\003J\r\202\212\203\261\034\300\252\321\273\233\321\221\334\032" signing_key_seq_num: 0 } master_addrs { host: "127.3.15.190" port: 34417 } master_addrs { host: "127.3.15.189" port: 41941 } master_addrs { host: "127.3.15.188" port: 39851 } ) from 127.3.15.190:34417 11:18:15.153 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.152119 3840 raft_consensus.cc:472] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 11:18:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.152257 3840 raft_consensus.cc:494] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.152768 3840 leader_election.cc:296] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105), 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) 11:18:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.153318 3707 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "1734b9064c4943339961de5d872e236b" candidate_uuid: "73918a1b139243ff837530f26d7808c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" is_pre_election: true 11:18:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.153342 3440 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "1734b9064c4943339961de5d872e236b" candidate_uuid: "73918a1b139243ff837530f26d7808c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" is_pre_election: true 11:18:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.153403 3707 raft_consensus.cc:2307] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 73918a1b139243ff837530f26d7808c9 for term 1 because replica is either leader or believes a valid leader to be alive. 11:18:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.153473 3440 raft_consensus.cc:2276] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 73918a1b139243ff837530f26d7808c9 in current term 1: Already voted for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in this term. 11:18:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.154083 3509 leader_election.cc:310] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73918a1b139243ff837530f26d7808c9; no voters: 3ac72b4ba12248ddad09193985a1e867, 7585fc19f9934f8a9df6ecb0d9e664bd 11:18:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.154335 3840 raft_consensus.cc:2886] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Advancing to term 1 11:18:15.155 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.155423 3840 raft_consensus.cc:2588] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority 11:18:15.157 [DEBUG - New I/O worker #3] (Negotiator.java:764) Authenticated connection [id: 0x5c2b32c4, /127.0.0.1:44218 :> /127.3.15.189:41941] using SASL/GSSAPI 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:15.157910 3842 negotiation.cc:313] Failed RPC negotiation. Trace: 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.087843 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:44218 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.087991 (+ 148us) server_negotiation.cc:184] Beginning negotiation 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.087995 (+ 4us) server_negotiation.cc:373] Waiting for connection header 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.088007 (+ 12us) server_negotiation.cc:381] Connection header received 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.090707 (+ 2700us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.090711 (+ 4us) server_negotiation.cc:420] Received NEGOTIATE request from client 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.090763 (+ 52us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.090826 (+ 63us) server_negotiation.cc:205] Negotiated authn=SASL 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.093704 (+ 2878us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.093861 (+ 157us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.096186 (+ 2325us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.096754 (+ 568us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.096828 (+ 74us) server_negotiation.cc:589] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.100489 (+ 3661us) server_negotiation.cc:337] Received SASL_INITIATE NegotiatePB request 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.100513 (+ 24us) server_negotiation.cc:746] Received SASL_INITIATE request from client 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.100517 (+ 4us) server_negotiation.cc:757] Client requested to use mechanism: GSSAPI 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.100560 (+ 43us) server_negotiation.cc:786] Calling sasl_server_start() 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.102174 (+ 1614us) server_negotiation.cc:349] Sending SASL_CHALLENGE NegotiatePB response 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.111101 (+ 8927us) server_negotiation.cc:337] Received SASL_RESPONSE NegotiatePB request 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.111106 (+ 5us) server_negotiation.cc:820] Received SASL_RESPONSE request from client 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.111108 (+ 2us) server_negotiation.cc:830] Calling sasl_server_step() 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.111285 (+ 177us) server_negotiation.cc:349] Sending SASL_SUCCESS NegotiatePB response 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.111426 (+ 141us) server_negotiation.cc:890] Waiting for connection context 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:15.157280 (+ 45854us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44218: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:44218): Connection reset by peer (error 104) 11:18:15.158 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":47} 11:18:15.160 [DEBUG - New I/O worker #6] (ConnectToCluster.java:331) Callback already invoked, discarding response(role: LEADER ca_cert_der: "0\202\001\3170\202\0018\240\003\002\001\002\002\t\000\256\311\337\365z\322\26540\r\006\t*\206H\206\367\r\001\001\v\005\0000\0271\0250\023\006\003U\004\003\023\fkudu-ipki-ca0\036\027\r190406111809Z\027\r290403111809Z0\0271\0250\023\006\003U\004\003\023\fkudu-ipki-ca0\201\2370\r\006\t*\206H\206\367\r\001\001\001\005\000\003\201\215\0000\201\211\002\201\201\000\275<\341\223\304\316\333D\353\323\334\004\346;\372\323i\3144\206\024\214dP\2272/4\327k\214\362)\004\363\254u\001\235\230\tA\005\224\367\222\035\003\213\244\371\241\031T4\264\321\325\376*\327\217_\3571{\221\301\223\036J\341m=\3363\251q\21535r\030\253Vh}9:\256\200\246\225\212\220\230\366\302X\317\301\036\202Ey0\341\334\232b\252\355\231{T\316\317\2475\237\267W\f\344o\216\336\261\002\003\001\000\001\243#0!0\016\006\003U\035\017\001\001\377\004\004\003\002\002\0040\017\006\003U\035\023\001\001\377\004\0050\003\001\001\3770\r\006\t*\206H\206\367\r\001\001\v\005\000\003\201\201\000\213\206\305\253\355,\2264\r1\353\002KN\376\341\216Rk\310\324U_\247\325\305\2067\3016J\313\366\376\336\006\246\376\274wr\323\316>!\003\021+\324xl\316OG#\022\017\364L\314\361<\037U\364\353\333\ff\216\001\222\353\300C\326\265\275\274\224\256\246~+H\211+KH|\347\270J\374\342\247\245\024\335\357\233V\203N\230\362\016\031\310\342\277\270\340S\262\234R[}\376\340R\356V\366@\304(" authn_token { token_data: "\b\370\225\242\345\005\032\f\n\ntest-admin" signature: "C\021\312\2200\257-XIB\021\271;n\342\304sm5\322?\241C\277W\200\217\304\301X\260\037p\021\355\034\003aG?\337\tR\2176Tg\003J\r\202\212\203\261\034\300\252\321\273\233\321\221\334\032" signing_key_seq_num: 0 } master_addrs { host: "127.3.15.190" port: 34417 } master_addrs { host: "127.3.15.189" port: 41941 } master_addrs { host: "127.3.15.188" port: 39851 } ) from 127.3.15.190:34417 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.160770 3241 catalog_manager.cc:1415] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44224: 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "TestAuthTokenReacquire-table-0" 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:15.161 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 11:18:15.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:15.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:15.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:15.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:15.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:15.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:15.161170 3241 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAuthTokenReacquire-table-0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 11:18:15.168 [DEBUG - New I/O worker #6] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:15.169 [DEBUG - New I/O worker #6] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=Kudu Master, attempt=4, TimeoutTracker(timeout=50000, elapsed=146), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [21ms] received response from server master-127.3.15.189:41941: Network error: connection disconnected, [21ms] delaying RPC due to: Network error: connection disconnected, [46ms] refreshing cache from master, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [46ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [62ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [64ms] Sub RPC ConnectToMaster: waiting for new authn token, [71ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [72ms] sending RPC to server master-127.3.15.189:41941, [113ms] received response from server master-127.3.15.189:41941: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [120ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [130ms] delaying RPC due to: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [134ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, [142ms] refreshing cache from master, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [143ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [144ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [144ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [145ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.190:34417: OK, [145ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=Deferred@1832732163(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduClient$6@1e63ce9d -> (continuation of Deferred@1821986742 after org.apache.kudu.client.AsyncKuduClient$1@1b75fceb@460717291) -> (continuation of Deferred@1471812961 after retry RPC@637208736), errback=passthrough -> (continuation of Deferred@1821986742 after org.apache.kudu.client.AsyncKuduClient$1@1b75fceb@460717291) -> (continuation of Deferred@1471812961 after retry RPC@637208736))) after lookup 11:18:15.171 [DEBUG - New I/O worker #6] (AsyncKuduClient.java:791) Opened table 3b699b77de1f4f1a8674230cc14f7631 11:18:15.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.177428 3811 raft_consensus.cc:472] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 11:18:15.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.177628 3811 raft_consensus.cc:494] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.177999 3811 leader_election.cc:296] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203), 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) 11:18:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.178607 3440 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "8a123165cc324cbe9ea0249a3891a94b" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" is_pre_election: true 11:18:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.178611 3575 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "8a123165cc324cbe9ea0249a3891a94b" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73918a1b139243ff837530f26d7808c9" is_pre_election: true 11:18:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.178712 3575 raft_consensus.cc:2346] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 0. 11:18:15.178 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.178712 3440 raft_consensus.cc:2346] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 0. 11:18:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.179059 3638 leader_election.cc:310] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 7585fc19f9934f8a9df6ecb0d9e664bd; no voters: 11:18:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.179258 3811 raft_consensus.cc:2640] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Leader pre-election won for term 1 11:18:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.179287 3811 raft_consensus.cc:472] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 11:18:15.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.179302 3811 raft_consensus.cc:2886] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Advancing to term 1 11:18:15.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.180168 3811 raft_consensus.cc:494] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.180426 3811 leader_election.cc:296] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 election: Requested vote from peers 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203), 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) 11:18:15.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.180898 3440 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "8a123165cc324cbe9ea0249a3891a94b" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ac72b4ba12248ddad09193985a1e867" 11:18:15.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.180966 3440 raft_consensus.cc:2886] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Advancing to term 1 11:18:15.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.181051 3575 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "8a123165cc324cbe9ea0249a3891a94b" candidate_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73918a1b139243ff837530f26d7808c9" 11:18:15.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.181102 3575 raft_consensus.cc:2886] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Advancing to term 1 11:18:15.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.181877 3440 raft_consensus.cc:2346] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 1. 11:18:15.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.181891 3575 raft_consensus.cc:2346] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7585fc19f9934f8a9df6ecb0d9e664bd in term 1. 11:18:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.182205 3638 leader_election.cc:310] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 7585fc19f9934f8a9df6ecb0d9e664bd; no voters: 11:18:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.182379 3811 raft_consensus.cc:2640] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Leader election won for term 1 11:18:15.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.182462 3811 raft_consensus.cc:667] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 LEADER]: Becoming Leader. State: Replica: 7585fc19f9934f8a9df6ecb0d9e664bd, State: Running, Role: LEADER 11:18:15.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.182641 3811 consensus_queue.cc:206] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.184118 3241 catalog_manager.cc:3976] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd reported cstate change: term changed from 0 to 1, leader changed from to 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131). New cstate: current_term: 1 leader_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } health_report { overall_health: HEALTHY } } } 11:18:15.207 [DEBUG - New I/O worker #4] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:15.208 [DEBUG - New I/O worker #6] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:15.208 [DEBUG - New I/O worker #5] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:15.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.212571 3575 raft_consensus.cc:1181] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Refusing update from remote peer 7585fc19f9934f8a9df6ecb0d9e664bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.212952 3440 raft_consensus.cc:1181] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Refusing update from remote peer 7585fc19f9934f8a9df6ecb0d9e664bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.213349 3811 consensus_queue.cc:984] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.213569 3815 consensus_queue.cc:984] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.228911 3440 raft_consensus.cc:1181] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Refusing update from remote peer 7585fc19f9934f8a9df6ecb0d9e664bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.229349 3811 consensus_queue.cc:984] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.232952 3575 raft_consensus.cc:1181] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Refusing update from remote peer 7585fc19f9934f8a9df6ecb0d9e664bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.233867 3815 consensus_queue.cc:984] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.274900 3440 raft_consensus.cc:1181] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Refusing update from remote peer 73918a1b139243ff837530f26d7808c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.275477 3840 consensus_queue.cc:984] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.278589 3707 raft_consensus.cc:1181] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Refusing update from remote peer 73918a1b139243ff837530f26d7808c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.279168 3818 consensus_queue.cc:984] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.287134 3846 raft_consensus.cc:472] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 11:18:15.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.287240 3846 raft_consensus.cc:494] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.287518 3846 leader_election.cc:296] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203), 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) 11:18:15.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.287883 3707 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "3abd1116f6db4d80b285597f66adbb17" candidate_uuid: "3ac72b4ba12248ddad09193985a1e867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" is_pre_election: true 11:18:15.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.287951 3707 raft_consensus.cc:2346] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ac72b4ba12248ddad09193985a1e867 in term 0. 11:18:15.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.287950 3575 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "3abd1116f6db4d80b285597f66adbb17" candidate_uuid: "3ac72b4ba12248ddad09193985a1e867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73918a1b139243ff837530f26d7808c9" is_pre_election: true 11:18:15.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.288007 3575 raft_consensus.cc:2346] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ac72b4ba12248ddad09193985a1e867 in term 0. 11:18:15.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.288229 3372 leader_election.cc:310] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 7585fc19f9934f8a9df6ecb0d9e664bd; no voters: 11:18:15.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.288373 3846 raft_consensus.cc:2640] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Leader pre-election won for term 1 11:18:15.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.288399 3846 raft_consensus.cc:472] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 11:18:15.288 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.288414 3846 raft_consensus.cc:2886] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [term 0 FOLLOWER]: Advancing to term 1 11:18:15.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.289268 3846 raft_consensus.cc:494] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.289463 3846 leader_election.cc:296] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [CANDIDATE]: Term 1 election: Requested vote from peers 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203), 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) 11:18:15.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.289835 3707 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "3abd1116f6db4d80b285597f66adbb17" candidate_uuid: "3ac72b4ba12248ddad09193985a1e867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" 11:18:15.289 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.289891 3575 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "3abd1116f6db4d80b285597f66adbb17" candidate_uuid: "3ac72b4ba12248ddad09193985a1e867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73918a1b139243ff837530f26d7808c9" 11:18:15.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.289899 3707 raft_consensus.cc:2886] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 0 FOLLOWER]: Advancing to term 1 11:18:15.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.289929 3575 raft_consensus.cc:2886] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9 [term 0 FOLLOWER]: Advancing to term 1 11:18:15.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.290593 3575 raft_consensus.cc:2346] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac72b4ba12248ddad09193985a1e867 in term 1. 11:18:15.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.290596 3707 raft_consensus.cc:2346] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac72b4ba12248ddad09193985a1e867 in term 1. 11:18:15.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.290863 3372 leader_election.cc:310] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ac72b4ba12248ddad09193985a1e867, 73918a1b139243ff837530f26d7808c9; no voters: 11:18:15.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.291007 3846 raft_consensus.cc:2640] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Leader election won for term 1 11:18:15.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.291097 3846 raft_consensus.cc:667] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [term 1 LEADER]: Becoming Leader. State: Replica: 3ac72b4ba12248ddad09193985a1e867, State: Running, Role: LEADER 11:18:15.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.291237 3846 consensus_queue.cc:206] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } } 11:18:15.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.292651 3245 catalog_manager.cc:3976] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 reported cstate change: term changed from 0 to 1, leader changed from to 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129). New cstate: current_term: 1 leader_uuid: "3ac72b4ba12248ddad09193985a1e867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 } health_report { overall_health: UNKNOWN } } } 11:18:15.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.348947 3440 raft_consensus.cc:1181] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Refusing update from remote peer 73918a1b139243ff837530f26d7808c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.349560 3818 consensus_queue.cc:984] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.352 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.352604 3707 raft_consensus.cc:1181] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Refusing update from remote peer 73918a1b139243ff837530f26d7808c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.353101 3840 consensus_queue.cc:984] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.357873 3440 raft_consensus.cc:1181] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Refusing update from remote peer 7585fc19f9934f8a9df6ecb0d9e664bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.358319 3815 consensus_queue.cc:984] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.361330 3575 raft_consensus.cc:1181] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Refusing update from remote peer 7585fc19f9934f8a9df6ecb0d9e664bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.361840 3815 consensus_queue.cc:984] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.438510 3707 raft_consensus.cc:1181] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Refusing update from remote peer 3ac72b4ba12248ddad09193985a1e867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.439172 3846 consensus_queue.cc:984] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.443274 3575 raft_consensus.cc:1181] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Refusing update from remote peer 3ac72b4ba12248ddad09193985a1e867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.443686 3816 consensus_queue.cc:984] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.593096 3440 raft_consensus.cc:1181] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Refusing update from remote peer 7585fc19f9934f8a9df6ecb0d9e664bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.593 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.593688 3815 consensus_queue.cc:984] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ac72b4ba12248ddad09193985a1e867" member_type: VOTER last_known_addr { host: "127.3.15.129" port: 33105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.596935 3575 raft_consensus.cc:1181] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Refusing update from remote peer 7585fc19f9934f8a9df6ecb0d9e664bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.597429 3815 consensus_queue.cc:984] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.723664 3575 raft_consensus.cc:1181] T 3abd1116f6db4d80b285597f66adbb17 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Refusing update from remote peer 3ac72b4ba12248ddad09193985a1e867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.724241 3816 consensus_queue.cc:984] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73918a1b139243ff837530f26d7808c9" member_type: VOTER last_known_addr { host: "127.3.15.130" port: 37203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:15.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.728683 3707 raft_consensus.cc:1181] T 3abd1116f6db4d80b285597f66adbb17 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Refusing update from remote peer 3ac72b4ba12248ddad09193985a1e867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:15.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:15.729426 3816 consensus_queue.cc:984] T 3abd1116f6db4d80b285597f66adbb17 P 3ac72b4ba12248ddad09193985a1e867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7585fc19f9934f8a9df6ecb0d9e664bd" member_type: VOTER last_known_addr { host: "127.3.15.131" port: 44057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:16.860 [DEBUG - New I/O worker #7] (Negotiator.java:764) Authenticated connection [id: 0x40e66c91, /127.0.0.1:44226 => /127.3.15.189:41941] using TOKEN/null 11:18:16.862 [DEBUG - New I/O worker #7] (AsyncKuduClient.java:791) Opened table 3aa15bdd6f364b75a22e2b2be21a8a01 11:18:16.863 [DEBUG - New I/O worker #7] (AsyncKuduClient.java:791) Opened table d6d879c7542741c8a7be7786b888ddd3 11:18:16.864 [DEBUG - New I/O worker #7] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:16.915 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:16.914983 3877 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44228: invalid token 11:18:16.915 [DEBUG - New I/O worker #8] (Negotiator.java:291) peer /127.3.15.189:41941 sent connection negotiation error: Not authorized: invalid token 11:18:16.916 [DEBUG - New I/O worker #8] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:16.920 [DEBUG - New I/O worker #2] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.190 11:18:16.920 [DEBUG - New I/O worker #3] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.189 11:18:16.920 [DEBUG - New I/O worker #1] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.188 11:18:16.941 [DEBUG - New I/O worker #3] (Negotiator.java:764) Authenticated connection [id: 0xd3934ac9, /127.0.0.1:44234 => /127.3.15.189:41941] using SASL/GSSAPI 11:18:16.943 [DEBUG - New I/O worker #2] (Negotiator.java:764) Authenticated connection [id: 0xc101e232, /127.0.0.1:56260 => /127.3.15.190:34417] using SASL/GSSAPI 11:18:16.943 [INFO - New I/O worker #3] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:16.947 [DEBUG - New I/O worker #3] (AsyncKuduClient.java:791) Opened table d70654235532446a85c0dc05f79f3c4f 11:18:16.949 [DEBUG - New I/O worker #2] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:16.952 [DEBUG - New I/O worker #1] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state NEGOTIATING due to: connection disconnected 11:18:16.953 [INFO - New I/O worker #1] (ConnectToCluster.java:361) Unable to connect to master 127.3.15.188:39851: connection disconnected 11:18:16.953 [DEBUG - New I/O worker #3] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:16.954094 3878 negotiation.cc:313] Failed RPC negotiation. Trace: 11:18:16.954 [INFO - New I/O worker #3] (AsyncKuduClient.java:2058) Invalidating location master-127.3.15.189:41941(127.3.15.189:41941) for tablet Kudu Master: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.917054 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:37780 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.917489 (+ 435us) server_negotiation.cc:184] Beginning negotiation 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.917492 (+ 3us) server_negotiation.cc:373] Waiting for connection header 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.917505 (+ 13us) server_negotiation.cc:381] Connection header received 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.918268 (+ 763us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.918274 (+ 6us) server_negotiation.cc:420] Received NEGOTIATE request from client 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.918398 (+ 124us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.918782 (+ 384us) server_negotiation.cc:205] Negotiated authn=SASL 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.935020 (+ 16238us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.935521 (+ 501us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.938232 (+ 2711us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.938855 (+ 623us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.938943 (+ 88us) server_negotiation.cc:589] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.942953 (+ 4010us) server_negotiation.cc:337] Received SASL_INITIATE NegotiatePB request 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.942986 (+ 33us) server_negotiation.cc:746] Received SASL_INITIATE request from client 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.942990 (+ 4us) server_negotiation.cc:757] Client requested to use mechanism: GSSAPI 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.943071 (+ 81us) server_negotiation.cc:786] Calling sasl_server_start() 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.947093 (+ 4022us) server_negotiation.cc:349] Sending SASL_CHALLENGE NegotiatePB response 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:16.953557 (+ 6464us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37780: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:37780): Cannot send after transport endpoint shutdown (error 108) 11:18:16.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":86,"threads_started":1} 11:18:16.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:16.954496 3229 connection.cc:647] server connection from 127.0.0.1:44234 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44234): Connection reset by peer (error 104) 11:18:16.972 [DEBUG - New I/O worker #4] (Negotiator.java:764) Authenticated connection [id: 0xd1d8e048, /127.0.0.1:37786 => /127.3.15.188:39851] using TOKEN/null 11:18:16.974 [DEBUG - New I/O worker #6] (Negotiator.java:764) Authenticated connection [id: 0x74a534f3, /127.0.0.1:44240 => /127.3.15.189:41941] using TOKEN/null 11:18:16.975 [DEBUG - New I/O worker #6] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:16.976 [DEBUG - New I/O worker #6] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=GetTableSchema, tablet=Kudu Master, attempt=3, TimeoutTracker(timeout=50000, elapsed=65), Traces: [0ms] sending RPC to server master-127.3.15.189:41941, [8ms] waiting for new authn token, [34ms] sending RPC to server master-127.3.15.189:41941, [44ms] received response from server master-127.3.15.189:41941: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [44ms] delaying RPC due to: Network error: [peer master-127.3.15.189:41941(127.3.15.189:41941)] disconnected from peer, [55ms] refreshing cache from master, [56ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.188:39851, [56ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.190:34417, [56ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.15.189:41941, [64ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.188:39851: OK, [65ms] Sub RPC ConnectToMaster: received response from server master-127.3.15.189:41941: OK, deferred=Deferred@1802619933(state=PENDING, result=null, callback=org.apache.kudu.client.AsyncKuduClient$6@6c67cd0f -> wakeup thread Thread-6, errback=passthrough -> wakeup thread Thread-6)) after lookup 11:18:16.977 [DEBUG - New I/O worker #5] (Negotiator.java:764) Authenticated connection [id: 0xb7570ec2, /127.0.0.1:56266 => /127.3.15.190:34417] using TOKEN/null 11:18:16.977 [DEBUG - New I/O worker #6] (AsyncKuduClient.java:791) Opened table 329758112d6c49beb826456c5e22abca 11:18:16.980 [DEBUG - New I/O worker #5] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:16.981 [DEBUG - New I/O worker #6] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:16.981 [DEBUG - New I/O worker #4] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:17.038 [DEBUG - New I/O worker #7] (Negotiator.java:764) Authenticated connection [id: 0xfc88eff6, /127.0.0.1:44242 => /127.3.15.189:41941] using TOKEN/null 11:18:17.039 [DEBUG - New I/O worker #7] (AsyncKuduClient.java:791) Opened table 7cf5d84d15a24a11abea7c2f5bdd3a06 11:18:17.041 [DEBUG - New I/O worker #7] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:17.164 [DEBUG - New I/O worker #8] (Negotiator.java:764) Authenticated connection [id: 0xa7274fe5, /127.0.0.1:44244 => /127.3.15.189:41941] using TOKEN/null 11:18:17.165 [DEBUG - New I/O worker #8] (AsyncKuduClient.java:791) Opened table a45d2c9dd7fe4a04a35d322134f2a9fa 11:18:17.167 [DEBUG - New I/O worker #8] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:17.213 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:17.213907 3877 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44246: token signing key expired 11:18:17.214 [DEBUG - New I/O worker #1] (Negotiator.java:291) peer /127.3.15.189:41941 sent connection negotiation error: Not authorized: token signing key expired 11:18:17.214 [DEBUG - New I/O worker #1] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:17.216 [DEBUG - New I/O worker #2] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.188 11:18:17.219 [DEBUG - New I/O worker #4] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.189 11:18:17.219 [DEBUG - New I/O worker #3] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.190 11:18:17.229 [DEBUG - New I/O worker #4] (Negotiator.java:764) Authenticated connection [id: 0x63a57229, /127.0.0.1:44252 => /127.3.15.189:41941] using SASL/GSSAPI 11:18:17.230 [DEBUG - New I/O worker #3] (Negotiator.java:764) Authenticated connection [id: 0x10340025, /127.0.0.1:56278 => /127.3.15.190:34417] using SASL/GSSAPI 11:18:17.231 [DEBUG - New I/O worker #2] (Negotiator.java:764) Authenticated connection [id: 0xecdaa261, /127.0.0.1:37798 => /127.3.15.188:39851] using SASL/GSSAPI 11:18:17.231 [INFO - New I/O worker #4] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:17.233 [DEBUG - New I/O worker #4] (AsyncKuduClient.java:791) Opened table 3b699b77de1f4f1a8674230cc14f7631 11:18:17.233 [DEBUG - New I/O worker #3] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:17.233 [DEBUG - New I/O worker #2] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:17.234 [DEBUG - New I/O worker #4] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:18.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:18.873044 3882 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44254: token expired 11:18:18.873 [DEBUG - New I/O worker #5] (Negotiator.java:291) peer /127.3.15.189:41941 sent connection negotiation error: Not authorized: token expired 11:18:18.873 [DEBUG - New I/O worker #5] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:18.876 [DEBUG - New I/O worker #7] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.190 11:18:18.876 [DEBUG - New I/O worker #8] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.189 11:18:18.877 [DEBUG - New I/O worker #6] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.15.188 11:18:18.890 [DEBUG - New I/O worker #7] (Negotiator.java:764) Authenticated connection [id: 0x07f9cce9, /127.0.0.1:56286 => /127.3.15.190:34417] using SASL/GSSAPI 11:18:18.891 [DEBUG - New I/O worker #6] (Negotiator.java:764) Authenticated connection [id: 0x3d3f9594, /127.0.0.1:37806 => /127.3.15.188:39851] using SASL/GSSAPI 11:18:18.896 [DEBUG - New I/O worker #8] (Negotiator.java:764) Authenticated connection [id: 0xc19a951f, /127.0.0.1:44260 => /127.3.15.189:41941] using SASL/GSSAPI 11:18:18.898 [INFO - New I/O worker #8] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:18.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.899399 3245 catalog_manager.cc:1889] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44260: 11:18:18.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) table { table_name: "TestAuthTokenReacquire-table-7" } 11:18:18.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.899422 3241 catalog_manager.cc:1889] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44260: 11:18:18.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) table { table_name: "TestAuthTokenReacquire-table-1" } 11:18:18.907 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.907368 3241 catalog_manager.cc:4215] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: Sending DeleteTablet for 3 replicas of tablet 1734b9064c4943339961de5d872e236b 11:18:18.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.907522 3241 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1734b9064c4943339961de5d872e236b on 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) (Table deleted at 2019-04-06 11:18:18 UTC) 11:18:18.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.907660 3241 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1734b9064c4943339961de5d872e236b on 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) (Table deleted at 2019-04-06 11:18:18 UTC) 11:18:18.908 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.907716 3241 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1734b9064c4943339961de5d872e236b on 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) (Table deleted at 2019-04-06 11:18:18 UTC) 11:18:18.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.908784 3554 tablet_service.cc:1047] Processing DeleteTablet for tablet 1734b9064c4943339961de5d872e236b with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:18 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:54644 11:18:18.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.908790 3687 tablet_service.cc:1047] Processing DeleteTablet for tablet 1734b9064c4943339961de5d872e236b with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:18 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:35738 11:18:18.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.908864 3419 tablet_service.cc:1047] Processing DeleteTablet for tablet 1734b9064c4943339961de5d872e236b with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:18 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:40778 11:18:18.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.909448 3897 tablet_replica.cc:264] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd: stopping tablet replica 11:18:18.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.909591 3899 tablet_replica.cc:264] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867: stopping tablet replica 11:18:18.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.909791 3898 tablet_replica.cc:264] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9: stopping tablet replica 11:18:18.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.909999 3899 raft_consensus.cc:2143] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:18.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.910164 3897 raft_consensus.cc:2143] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 LEADER]: Raft consensus shutting down. 11:18:18.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.910331 3899 raft_consensus.cc:2170] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:18.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.910552 3898 raft_consensus.cc:2143] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:18.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.910800 3245 catalog_manager.cc:4215] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: Sending DeleteTablet for 3 replicas of tablet 3337cdf76ae447bd9df561914cede569 11:18:18.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.910869 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3337cdf76ae447bd9df561914cede569 on 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) (Table deleted at 2019-04-06 11:18:18 UTC) 11:18:18.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.910944 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3337cdf76ae447bd9df561914cede569 on 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) (Table deleted at 2019-04-06 11:18:18 UTC) 11:18:18.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.910995 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 3337cdf76ae447bd9df561914cede569 on 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) (Table deleted at 2019-04-06 11:18:18 UTC) 11:18:18.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.911633 3419 tablet_service.cc:1047] Processing DeleteTablet for tablet 3337cdf76ae447bd9df561914cede569 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:18 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:40778 11:18:18.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.911741 3899 ts_tablet_manager.cc:1377] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:18.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.911913 3687 tablet_service.cc:1047] Processing DeleteTablet for tablet 3337cdf76ae447bd9df561914cede569 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:18 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:35738 11:18:18.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.913512 3897 raft_consensus.cc:2170] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:18.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.911526 3554 tablet_service.cc:1047] Processing DeleteTablet for tablet 3337cdf76ae447bd9df561914cede569 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:18 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:54644 11:18:18.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.914438 3898 raft_consensus.cc:2170] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:18.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.916133 3898 ts_tablet_manager.cc:1377] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:18.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.916738 3899 ts_tablet_manager.cc:1390] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:18.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.916795 3899 log.cc:990] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal/wals/1734b9064c4943339961de5d872e236b 11:18:18.917 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.917065 3899 ts_tablet_manager.cc:1411] T 1734b9064c4943339961de5d872e236b P 3ac72b4ba12248ddad09193985a1e867: Deleting consensus metadata 11:18:18.918 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.918287 3898 ts_tablet_manager.cc:1390] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:18.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.919462 3897 ts_tablet_manager.cc:1377] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:18.919 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.919884 3226 catalog_manager.cc:3356] TS 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105): tablet 1734b9064c4943339961de5d872e236b (table TestAuthTokenReacquire-table-1 [id=3aa15bdd6f364b75a22e2b2be21a8a01]) successfully deleted 11:18:18.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.920133 3899 tablet_replica.cc:264] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867: stopping tablet replica 11:18:18.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.920337 3899 raft_consensus.cc:2143] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:18.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.920466 3899 raft_consensus.cc:2170] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:18.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.920979 3899 ts_tablet_manager.cc:1377] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:18.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.921635 3897 ts_tablet_manager.cc:1390] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:18.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.921684 3897 log.cc:990] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal/wals/1734b9064c4943339961de5d872e236b 11:18:18.921 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.921903 3897 ts_tablet_manager.cc:1411] T 1734b9064c4943339961de5d872e236b P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting consensus metadata 11:18:18.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.922811 3899 ts_tablet_manager.cc:1390] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:18.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.922889 3897 tablet_replica.cc:264] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd: stopping tablet replica 11:18:18.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.922976 3897 raft_consensus.cc:2143] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 LEADER]: Raft consensus shutting down. 11:18:18.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.922988 3227 catalog_manager.cc:3356] TS 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057): tablet 1734b9064c4943339961de5d872e236b (table TestAuthTokenReacquire-table-1 [id=3aa15bdd6f364b75a22e2b2be21a8a01]) successfully deleted 11:18:18.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.922996 3899 log.cc:990] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal/wals/3337cdf76ae447bd9df561914cede569 11:18:18.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.923220 3899 ts_tablet_manager.cc:1411] T 3337cdf76ae447bd9df561914cede569 P 3ac72b4ba12248ddad09193985a1e867: Deleting consensus metadata 11:18:18.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.924038 3226 catalog_manager.cc:3356] TS 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105): tablet 3337cdf76ae447bd9df561914cede569 (table TestAuthTokenReacquire-table-7 [id=d6d879c7542741c8a7be7786b888ddd3]) successfully deleted 11:18:18.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.924676 3898 log.cc:990] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal/wals/1734b9064c4943339961de5d872e236b 11:18:18.924 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.924902 3898 ts_tablet_manager.cc:1411] T 1734b9064c4943339961de5d872e236b P 73918a1b139243ff837530f26d7808c9: Deleting consensus metadata 11:18:18.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.926017 3898 tablet_replica.cc:264] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9: stopping tablet replica 11:18:18.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.926095 3227 catalog_manager.cc:3356] TS 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203): tablet 1734b9064c4943339961de5d872e236b (table TestAuthTokenReacquire-table-1 [id=3aa15bdd6f364b75a22e2b2be21a8a01]) successfully deleted 11:18:18.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.926271 3897 raft_consensus.cc:2170] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:18.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.926628 3897 ts_tablet_manager.cc:1377] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:18.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.927896 3897 ts_tablet_manager.cc:1390] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:18.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.927934 3897 log.cc:990] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal/wals/3337cdf76ae447bd9df561914cede569 11:18:18.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.928140 3897 ts_tablet_manager.cc:1411] T 3337cdf76ae447bd9df561914cede569 P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting consensus metadata 11:18:18.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.928740 3898 raft_consensus.cc:2143] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:18.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.928846 3227 catalog_manager.cc:3356] TS 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057): tablet 3337cdf76ae447bd9df561914cede569 (table TestAuthTokenReacquire-table-7 [id=d6d879c7542741c8a7be7786b888ddd3]) successfully deleted 11:18:18.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.928932 3898 raft_consensus.cc:2170] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:18.929 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.929332 3898 ts_tablet_manager.cc:1377] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:18.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.930640 3898 ts_tablet_manager.cc:1390] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:18.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.930675 3898 log.cc:990] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal/wals/3337cdf76ae447bd9df561914cede569 11:18:18.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.930810 3898 ts_tablet_manager.cc:1411] T 3337cdf76ae447bd9df561914cede569 P 73918a1b139243ff837530f26d7808c9: Deleting consensus metadata 11:18:18.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.931618 3227 catalog_manager.cc:3356] TS 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203): tablet 3337cdf76ae447bd9df561914cede569 (table TestAuthTokenReacquire-table-7 [id=d6d879c7542741c8a7be7786b888ddd3]) successfully deleted 11:18:18.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.956156 3245 catalog_manager.cc:1889] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44260: 11:18:18.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) table { table_name: "TestAuthTokenReacquire-table-6" } 11:18:18.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.961057 3245 catalog_manager.cc:4215] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: Sending DeleteTablet for 3 replicas of tablet c07b620efb024973aea4f313eabf4592 11:18:18.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.961158 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c07b620efb024973aea4f313eabf4592 on 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) (Table deleted at 2019-04-06 11:18:18 UTC) 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.961244 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c07b620efb024973aea4f313eabf4592 on 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) (Table deleted at 2019-04-06 11:18:18 UTC) 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.961298 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c07b620efb024973aea4f313eabf4592 on 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) (Table deleted at 2019-04-06 11:18:18 UTC) 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.961879 3554 tablet_service.cc:1047] Processing DeleteTablet for tablet c07b620efb024973aea4f313eabf4592 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:18 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:54644 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.961993 3687 tablet_service.cc:1047] Processing DeleteTablet for tablet c07b620efb024973aea4f313eabf4592 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:18 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:35738 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.962113 3898 tablet_replica.cc:264] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9: stopping tablet replica 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.962148 3897 tablet_replica.cc:264] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd: stopping tablet replica 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.962241 3897 raft_consensus.cc:2143] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 LEADER]: Raft consensus shutting down. 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.962245 3898 raft_consensus.cc:2143] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.962330 3898 raft_consensus.cc:2170] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.962370 3897 raft_consensus.cc:2170] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.962525 3419 tablet_service.cc:1047] Processing DeleteTablet for tablet c07b620efb024973aea4f313eabf4592 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:18 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:40778 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.962857 3897 ts_tablet_manager.cc:1377] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:18.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.962888 3898 ts_tablet_manager.cc:1377] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:18.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.964946 3898 ts_tablet_manager.cc:1390] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:18.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.965186 3898 log.cc:990] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal/wals/c07b620efb024973aea4f313eabf4592 11:18:18.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.965348 3897 ts_tablet_manager.cc:1390] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:18.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.965504 3897 log.cc:990] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal/wals/c07b620efb024973aea4f313eabf4592 11:18:18.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.965509 3898 ts_tablet_manager.cc:1411] T c07b620efb024973aea4f313eabf4592 P 73918a1b139243ff837530f26d7808c9: Deleting consensus metadata 11:18:18.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.965809 3899 tablet_replica.cc:264] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867: stopping tablet replica 11:18:18.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.965836 3897 ts_tablet_manager.cc:1411] T c07b620efb024973aea4f313eabf4592 P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting consensus metadata 11:18:18.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.966097 3899 raft_consensus.cc:2143] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:18.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.966303 3899 raft_consensus.cc:2170] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:18.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.966363 3227 catalog_manager.cc:3356] TS 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203): tablet c07b620efb024973aea4f313eabf4592 (table TestAuthTokenReacquire-table-6 [id=d70654235532446a85c0dc05f79f3c4f]) successfully deleted 11:18:18.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.966466 3227 catalog_manager.cc:3356] TS 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057): tablet c07b620efb024973aea4f313eabf4592 (table TestAuthTokenReacquire-table-6 [id=d70654235532446a85c0dc05f79f3c4f]) successfully deleted 11:18:18.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.966948 3899 ts_tablet_manager.cc:1377] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:18.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.968406 3899 ts_tablet_manager.cc:1390] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:18.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.968439 3899 log.cc:990] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal/wals/c07b620efb024973aea4f313eabf4592 11:18:18.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.968608 3899 ts_tablet_manager.cc:1411] T c07b620efb024973aea4f313eabf4592 P 3ac72b4ba12248ddad09193985a1e867: Deleting consensus metadata 11:18:18.969 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.969440 3226 catalog_manager.cc:3356] TS 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105): tablet c07b620efb024973aea4f313eabf4592 (table TestAuthTokenReacquire-table-6 [id=d70654235532446a85c0dc05f79f3c4f]) successfully deleted 11:18:18.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.981387 3245 catalog_manager.cc:1889] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44260: 11:18:18.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) table { table_name: "TestAuthTokenReacquire-table-4" } 11:18:18.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.985479 3245 catalog_manager.cc:4215] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: Sending DeleteTablet for 3 replicas of tablet fe1a243ef4c64777b0a73ede1d7f653a 11:18:18.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.985563 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fe1a243ef4c64777b0a73ede1d7f653a on 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) (Table deleted at 2019-04-06 11:18:18 UTC) 11:18:18.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.985656 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fe1a243ef4c64777b0a73ede1d7f653a on 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) (Table deleted at 2019-04-06 11:18:18 UTC) 11:18:18.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.985713 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet fe1a243ef4c64777b0a73ede1d7f653a on 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) (Table deleted at 2019-04-06 11:18:18 UTC) 11:18:18.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.986220 3687 tablet_service.cc:1047] Processing DeleteTablet for tablet fe1a243ef4c64777b0a73ede1d7f653a with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:18 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:35738 11:18:18.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.986639 3554 tablet_service.cc:1047] Processing DeleteTablet for tablet fe1a243ef4c64777b0a73ede1d7f653a with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:18 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:54644 11:18:18.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.986836 3898 tablet_replica.cc:264] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9: stopping tablet replica 11:18:18.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.986948 3898 raft_consensus.cc:2143] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 1 LEADER]: Raft consensus shutting down. 11:18:18.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.987123 3419 tablet_service.cc:1047] Processing DeleteTablet for tablet fe1a243ef4c64777b0a73ede1d7f653a with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:18 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:40778 11:18:18.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.987205 3898 raft_consensus.cc:2170] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:18.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.987787 3898 ts_tablet_manager.cc:1377] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:18.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.989899 3898 ts_tablet_manager.cc:1390] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:18.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.990064 3898 log.cc:990] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal/wals/fe1a243ef4c64777b0a73ede1d7f653a 11:18:18.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.990394 3898 ts_tablet_manager.cc:1411] T fe1a243ef4c64777b0a73ede1d7f653a P 73918a1b139243ff837530f26d7808c9: Deleting consensus metadata 11:18:18.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.990914 3899 tablet_replica.cc:264] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867: stopping tablet replica 11:18:18.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.991055 3899 raft_consensus.cc:2143] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:18.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.991142 3899 raft_consensus.cc:2170] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:18.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.991472 3227 catalog_manager.cc:3356] TS 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203): tablet fe1a243ef4c64777b0a73ede1d7f653a (table TestAuthTokenReacquire-table-4 [id=329758112d6c49beb826456c5e22abca]) successfully deleted 11:18:18.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.991658 3899 ts_tablet_manager.cc:1377] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:18.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.992728 3897 tablet_replica.cc:264] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd: stopping tablet replica 11:18:18.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.992843 3897 raft_consensus.cc:2143] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:18.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.992925 3897 raft_consensus.cc:2170] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:18.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.993314 3897 ts_tablet_manager.cc:1377] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:18.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.994629 3897 ts_tablet_manager.cc:1390] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:18.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.994664 3897 log.cc:990] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal/wals/fe1a243ef4c64777b0a73ede1d7f653a 11:18:18.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.994863 3897 ts_tablet_manager.cc:1411] T fe1a243ef4c64777b0a73ede1d7f653a P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting consensus metadata 11:18:18.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.995512 3227 catalog_manager.cc:3356] TS 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057): tablet fe1a243ef4c64777b0a73ede1d7f653a (table TestAuthTokenReacquire-table-4 [id=329758112d6c49beb826456c5e22abca]) successfully deleted 11:18:18.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.996364 3899 ts_tablet_manager.cc:1390] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:18.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.996404 3899 log.cc:990] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal/wals/fe1a243ef4c64777b0a73ede1d7f653a 11:18:18.996 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.996709 3899 ts_tablet_manager.cc:1411] T fe1a243ef4c64777b0a73ede1d7f653a P 3ac72b4ba12248ddad09193985a1e867: Deleting consensus metadata 11:18:18.997 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:18.997382 3226 catalog_manager.cc:3356] TS 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105): tablet fe1a243ef4c64777b0a73ede1d7f653a (table TestAuthTokenReacquire-table-4 [id=329758112d6c49beb826456c5e22abca]) successfully deleted 11:18:19.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.042323 3245 catalog_manager.cc:1889] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44260: 11:18:19.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) table { table_name: "TestAuthTokenReacquire-table-2" } 11:18:19.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.046644 3245 catalog_manager.cc:4215] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: Sending DeleteTablet for 3 replicas of tablet a5ef36a8cc8249cb98791ad5c9f0ce26 11:18:19.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.046738 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a5ef36a8cc8249cb98791ad5c9f0ce26 on 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) (Table deleted at 2019-04-06 11:18:19 UTC) 11:18:19.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.046846 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a5ef36a8cc8249cb98791ad5c9f0ce26 on 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) (Table deleted at 2019-04-06 11:18:19 UTC) 11:18:19.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.046911 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet a5ef36a8cc8249cb98791ad5c9f0ce26 on 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) (Table deleted at 2019-04-06 11:18:19 UTC) 11:18:19.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.047467 3687 tablet_service.cc:1047] Processing DeleteTablet for tablet a5ef36a8cc8249cb98791ad5c9f0ce26 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:19 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:35738 11:18:19.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.047487 3554 tablet_service.cc:1047] Processing DeleteTablet for tablet a5ef36a8cc8249cb98791ad5c9f0ce26 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:19 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:54644 11:18:19.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.048179 3419 tablet_service.cc:1047] Processing DeleteTablet for tablet a5ef36a8cc8249cb98791ad5c9f0ce26 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:19 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:40778 11:18:19.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.048852 3899 tablet_replica.cc:264] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867: stopping tablet replica 11:18:19.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.049070 3899 raft_consensus.cc:2143] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:19.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.049263 3899 raft_consensus.cc:2170] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:19.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.049945 3899 ts_tablet_manager.cc:1377] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:19.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.050050 3897 tablet_replica.cc:264] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd: stopping tablet replica 11:18:19.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.050153 3897 raft_consensus.cc:2143] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:19.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.050375 3897 raft_consensus.cc:2170] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:19.050 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.050880 3897 ts_tablet_manager.cc:1377] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:19.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.051152 3898 tablet_replica.cc:264] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9: stopping tablet replica 11:18:19.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.051375 3898 raft_consensus.cc:2143] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 1 LEADER]: Raft consensus shutting down. 11:18:19.051 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.051565 3898 raft_consensus.cc:2170] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:19.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.052069 3898 ts_tablet_manager.cc:1377] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:19.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.052196 3899 ts_tablet_manager.cc:1390] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:19.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.052244 3899 log.cc:990] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal/wals/a5ef36a8cc8249cb98791ad5c9f0ce26 11:18:19.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.052438 3899 ts_tablet_manager.cc:1411] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 3ac72b4ba12248ddad09193985a1e867: Deleting consensus metadata 11:18:19.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.053295 3226 catalog_manager.cc:3356] TS 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105): tablet a5ef36a8cc8249cb98791ad5c9f0ce26 (table TestAuthTokenReacquire-table-2 [id=7cf5d84d15a24a11abea7c2f5bdd3a06]) successfully deleted 11:18:19.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.053997 3898 ts_tablet_manager.cc:1390] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:19.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.054049 3898 log.cc:990] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal/wals/a5ef36a8cc8249cb98791ad5c9f0ce26 11:18:19.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.054250 3898 ts_tablet_manager.cc:1411] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 73918a1b139243ff837530f26d7808c9: Deleting consensus metadata 11:18:19.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.054723 3897 ts_tablet_manager.cc:1390] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:19.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.054764 3897 log.cc:990] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal/wals/a5ef36a8cc8249cb98791ad5c9f0ce26 11:18:19.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.054956 3897 ts_tablet_manager.cc:1411] T a5ef36a8cc8249cb98791ad5c9f0ce26 P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting consensus metadata 11:18:19.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.054988 3227 catalog_manager.cc:3356] TS 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203): tablet a5ef36a8cc8249cb98791ad5c9f0ce26 (table TestAuthTokenReacquire-table-2 [id=7cf5d84d15a24a11abea7c2f5bdd3a06]) successfully deleted 11:18:19.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.055632 3227 catalog_manager.cc:3356] TS 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057): tablet a5ef36a8cc8249cb98791ad5c9f0ce26 (table TestAuthTokenReacquire-table-2 [id=7cf5d84d15a24a11abea7c2f5bdd3a06]) successfully deleted 11:18:19.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.168272 3245 catalog_manager.cc:1889] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44260: 11:18:19.168 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) table { table_name: "TestAuthTokenReacquire-table-3" } 11:18:19.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.172420 3245 catalog_manager.cc:4215] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: Sending DeleteTablet for 3 replicas of tablet c156fd8d4e8f436c97eed32d4b50e1cd 11:18:19.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.172508 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c156fd8d4e8f436c97eed32d4b50e1cd on 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) (Table deleted at 2019-04-06 11:18:19 UTC) 11:18:19.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.172621 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c156fd8d4e8f436c97eed32d4b50e1cd on 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) (Table deleted at 2019-04-06 11:18:19 UTC) 11:18:19.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.172690 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet c156fd8d4e8f436c97eed32d4b50e1cd on 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) (Table deleted at 2019-04-06 11:18:19 UTC) 11:18:19.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.173329 3554 tablet_service.cc:1047] Processing DeleteTablet for tablet c156fd8d4e8f436c97eed32d4b50e1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:19 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:54644 11:18:19.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.173494 3687 tablet_service.cc:1047] Processing DeleteTablet for tablet c156fd8d4e8f436c97eed32d4b50e1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:19 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:35738 11:18:19.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.173554 3898 tablet_replica.cc:264] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9: stopping tablet replica 11:18:19.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.173655 3897 tablet_replica.cc:264] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd: stopping tablet replica 11:18:19.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.173728 3898 raft_consensus.cc:2143] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:19.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.173748 3897 raft_consensus.cc:2143] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:19.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.173816 3897 raft_consensus.cc:2170] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:19.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.173825 3898 raft_consensus.cc:2170] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:19.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.174336 3897 ts_tablet_manager.cc:1377] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:19.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.174407 3898 ts_tablet_manager.cc:1377] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:19.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.174533 3419 tablet_service.cc:1047] Processing DeleteTablet for tablet c156fd8d4e8f436c97eed32d4b50e1cd with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:19 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:40778 11:18:19.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.174736 3899 tablet_replica.cc:264] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867: stopping tablet replica 11:18:19.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.174845 3899 raft_consensus.cc:2143] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 1 LEADER]: Raft consensus shutting down. 11:18:19.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.175141 3899 raft_consensus.cc:2170] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:19.175 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.175645 3899 ts_tablet_manager.cc:1377] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:19.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.176362 3898 ts_tablet_manager.cc:1390] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:19.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.176403 3898 log.cc:990] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal/wals/c156fd8d4e8f436c97eed32d4b50e1cd 11:18:19.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.176636 3898 ts_tablet_manager.cc:1411] T c156fd8d4e8f436c97eed32d4b50e1cd P 73918a1b139243ff837530f26d7808c9: Deleting consensus metadata 11:18:19.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.176775 3897 ts_tablet_manager.cc:1390] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:19.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.176815 3897 log.cc:990] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal/wals/c156fd8d4e8f436c97eed32d4b50e1cd 11:18:19.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.177000 3897 ts_tablet_manager.cc:1411] T c156fd8d4e8f436c97eed32d4b50e1cd P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting consensus metadata 11:18:19.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.177425 3227 catalog_manager.cc:3356] TS 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203): tablet c156fd8d4e8f436c97eed32d4b50e1cd (table TestAuthTokenReacquire-table-3 [id=a45d2c9dd7fe4a04a35d322134f2a9fa]) successfully deleted 11:18:19.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.177645 3899 ts_tablet_manager.cc:1390] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:19.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.177687 3899 log.cc:990] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal/wals/c156fd8d4e8f436c97eed32d4b50e1cd 11:18:19.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.177737 3227 catalog_manager.cc:3356] TS 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057): tablet c156fd8d4e8f436c97eed32d4b50e1cd (table TestAuthTokenReacquire-table-3 [id=a45d2c9dd7fe4a04a35d322134f2a9fa]) successfully deleted 11:18:19.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.177837 3899 ts_tablet_manager.cc:1411] T c156fd8d4e8f436c97eed32d4b50e1cd P 3ac72b4ba12248ddad09193985a1e867: Deleting consensus metadata 11:18:19.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.179071 3226 catalog_manager.cc:3356] TS 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105): tablet c156fd8d4e8f436c97eed32d4b50e1cd (table TestAuthTokenReacquire-table-3 [id=a45d2c9dd7fe4a04a35d322134f2a9fa]) successfully deleted 11:18:19.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.234894 3245 catalog_manager.cc:1889] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44260: 11:18:19.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) table { table_name: "TestAuthTokenReacquire-table-5" } 11:18:19.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.238936 3245 catalog_manager.cc:4215] T 00000000000000000000000000000000 P a04f897cb6ac4fe38cfe92cc0220fa7c: Sending DeleteTablet for 3 replicas of tablet 8a123165cc324cbe9ea0249a3891a94b 11:18:19.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.239038 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8a123165cc324cbe9ea0249a3891a94b on 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203) (Table deleted at 2019-04-06 11:18:19 UTC) 11:18:19.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.239120 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8a123165cc324cbe9ea0249a3891a94b on 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105) (Table deleted at 2019-04-06 11:18:19 UTC) 11:18:19.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.239195 3245 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 8a123165cc324cbe9ea0249a3891a94b on 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057) (Table deleted at 2019-04-06 11:18:19 UTC) 11:18:19.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.239742 3687 tablet_service.cc:1047] Processing DeleteTablet for tablet 8a123165cc324cbe9ea0249a3891a94b with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:19 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:35738 11:18:19.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.239946 3554 tablet_service.cc:1047] Processing DeleteTablet for tablet 8a123165cc324cbe9ea0249a3891a94b with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:19 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:54644 11:18:19.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.240003 3897 tablet_replica.cc:264] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd: stopping tablet replica 11:18:19.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.240051 3419 tablet_service.cc:1047] Processing DeleteTablet for tablet 8a123165cc324cbe9ea0249a3891a94b with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:19 UTC) from {username='kudu', principal='kudu/127.3.15.189@KRBTEST.COM'} at 127.0.0.1:40778 11:18:19.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.240114 3897 raft_consensus.cc:2143] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 LEADER]: Raft consensus shutting down. 11:18:19.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.240237 3899 tablet_replica.cc:264] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867: stopping tablet replica 11:18:19.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.240336 3899 raft_consensus.cc:2143] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:19.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.240408 3899 raft_consensus.cc:2170] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:19.240 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.240406 3897 raft_consensus.cc:2170] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:19.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.240725 3898 tablet_replica.cc:264] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9: stopping tablet replica 11:18:19.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.240839 3898 raft_consensus.cc:2143] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:19.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.240921 3898 raft_consensus.cc:2170] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:19.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.240929 3897 ts_tablet_manager.cc:1377] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:19.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.240931 3899 ts_tablet_manager.cc:1377] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:19.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.241459 3898 ts_tablet_manager.cc:1377] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:19.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.242868 3897 ts_tablet_manager.cc:1390] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:19.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.242897 3897 log.cc:990] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-2/wal/wals/8a123165cc324cbe9ea0249a3891a94b 11:18:19.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.243122 3897 ts_tablet_manager.cc:1411] T 8a123165cc324cbe9ea0249a3891a94b P 7585fc19f9934f8a9df6ecb0d9e664bd: Deleting consensus metadata 11:18:19.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.243367 3898 ts_tablet_manager.cc:1390] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:19.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.243405 3898 log.cc:990] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-1/wal/wals/8a123165cc324cbe9ea0249a3891a94b 11:18:19.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.243516 3899 ts_tablet_manager.cc:1390] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 11:18:19.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.243551 3899 log.cc:990] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster5062749384301942558/ts-0/wal/wals/8a123165cc324cbe9ea0249a3891a94b 11:18:19.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.243607 3898 ts_tablet_manager.cc:1411] T 8a123165cc324cbe9ea0249a3891a94b P 73918a1b139243ff837530f26d7808c9: Deleting consensus metadata 11:18:19.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.243762 3899 ts_tablet_manager.cc:1411] T 8a123165cc324cbe9ea0249a3891a94b P 3ac72b4ba12248ddad09193985a1e867: Deleting consensus metadata 11:18:19.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.243875 3227 catalog_manager.cc:3356] TS 7585fc19f9934f8a9df6ecb0d9e664bd (127.3.15.131:44057): tablet 8a123165cc324cbe9ea0249a3891a94b (table TestAuthTokenReacquire-table-5 [id=3b699b77de1f4f1a8674230cc14f7631]) successfully deleted 11:18:19.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.244706 3227 catalog_manager.cc:3356] TS 73918a1b139243ff837530f26d7808c9 (127.3.15.130:37203): tablet 8a123165cc324cbe9ea0249a3891a94b (table TestAuthTokenReacquire-table-5 [id=3b699b77de1f4f1a8674230cc14f7631]) successfully deleted 11:18:19.244 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.244722 3226 catalog_manager.cc:3356] TS 3ac72b4ba12248ddad09193985a1e867 (127.3.15.129:33105): tablet 8a123165cc324cbe9ea0249a3891a94b (table TestAuthTokenReacquire-table-5 [id=3b699b77de1f4f1a8674230cc14f7631]) successfully deleted 11:18:19.245 [ERROR - main] (TestAuthTokenReacquire.java:151) exception in thread 0: {} org.apache.kudu.client.NonRecoverableException: table TestAuthTokenReacquire-table-0 already exists with id 0ccfb0ed3bc24bcf8b70085648f640fc at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:402) at org.apache.kudu.client.KuduClient.createTable(KuduClient.java:119) at org.apache.kudu.client.TestAuthTokenReacquire$1.run(TestAuthTokenReacquire.java:128) at java.lang.Thread.run(Thread.java:748) Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace at org.apache.kudu.client.RpcProxy.dispatchMasterError(RpcProxy.java:387) at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:280) at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:60) at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:150) at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:146) at org.apache.kudu.client.Connection.messageReceived(Connection.java:390) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.apache.kudu.client.Connection.handleUpstream(Connection.java:238) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 11:18:19.249 [DEBUG - New I/O worker #6] (Connection.java:691) [peer master-127.3.15.188:39851(127.3.15.188:39851)] cleaning up while in state READY due to: connection disconnected 11:18:19.249 [DEBUG - New I/O worker #7] (Connection.java:691) [peer master-127.3.15.190:34417(127.3.15.190:34417)] cleaning up while in state READY due to: connection disconnected 11:18:19.249 [DEBUG - New I/O worker #8] (Connection.java:691) [peer master-127.3.15.189:41941(127.3.15.189:41941)] cleaning up while in state READY due to: connection disconnected 11:18:19.250 [DEBUG - main] (AsyncKuduClient.java:2373) Releasing all remaining resources 11:18:19.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.251930 3134 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver with pid 3351 11:18:19.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.257417 3134 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver with pid 3501 11:18:19.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.260879 3134 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver with pid 3630 11:18:19.264 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.264243 3134 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master with pid 3153 11:18:19.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.267645 3134 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master with pid 3218 11:18:19.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.271181 3134 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master with pid 3283 11:18:19.283 [ERROR - main] (RetryRule.java:197) org.apache.kudu.client.TestAuthTokenReacquire.testBasicMasterOperations: failed attempt 1 java.lang.AssertionError: test failed: unexpected errors at org.junit.Assert.fail(Assert.java:88) at org.apache.kudu.client.TestAuthTokenReacquire.testBasicMasterOperations(TestAuthTokenReacquire.java:153) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:195) at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:212) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77) at org.junit.runner.JUnitCore.main(JUnitCore.java:36) 11:18:19.284 [ERROR - main] (RetryRule.java:219) org.apache.kudu.client.TestAuthTokenReacquire.testBasicMasterOperations: giving up after 1 attempts E11:18:19.329 [INFO - main] (RandomUtils.java:46) Using random seed: 1554549499329 11:18:19.332 [INFO - main] (RetryRule.java:135) Creating RetryStatement with result reporter and retry count of 0 (no retries) .11:18:19.332 [INFO - main] (KuduTestHarness.java:139) Creating a new MiniKuduCluster... 11:18:19.332 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-taskpieVMq/test-tmp 11:18:19.333 [INFO - main] (KuduBinaryLocator.java:78) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/debug/bin 11:18:19.333 [INFO - main] (MiniKuduCluster.java:187) Starting process: [/tmp/dist-test-taskpieVMq/java/build/dist-test/../../../build/debug/bin/kudu, test, mini_cluster, --serialization=pb] 11:18:19.335 [DEBUG - main] (MiniKuduCluster.java:155) Request: create_cluster { num_masters: 3 num_tservers: 3 enable_kerberos: true cluster_root: "/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631" extra_master_flags: "--authn_token_validity_seconds=1" extra_master_flags: "--authz_token_validity_seconds=1" extra_master_flags: "--rpc_inject_invalid_authn_token_ratio=0.5" extra_tserver_flags: "--rpc_inject_invalid_authn_token_ratio=0.5" extra_tserver_flags: "--tserver_enforce_access_control=true" extra_tserver_flags: "--tserver_inject_invalid_authz_token_ratio=0.5" hms_mode: NONE mini_kdc_options { } } 11:18:19.405 [DEBUG - main] (MiniKuduCluster.java:165) Response: 11:18:19.405 [DEBUG - main] (MiniKuduCluster.java:155) Request: start_cluster { } 11:18:19.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Loading random data 11:18:19.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Initializing database '/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/principal' for realm 'KRBTEST.COM', 11:18:19.440 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) master key name 'K/M@KRBTEST.COM' 11:18:19.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:19 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): setting up network... 11:18:19.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:19 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) 11:18:19.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked 11:18:19.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 11:18:19.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:19 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): skipping unrecognized local address family 17 11:18:19.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:19 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): skipping unrecognized local address family 17 11:18:19.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:19 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): set up 1 sockets 11:18:19.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:19 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): commencing operation 11:18:19.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) krb5kdc: starting... 11:18:19.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:19.781826 3903 mini_kdc.cc:119] Time spent starting KDC: real 0.368s user 0.002s sys 0.004s 11:18:19.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy 11:18:19.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 11:18:19.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-admin@KRBTEST.COM" created. 11:18:19.807 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy 11:18:19.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 11:18:19.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "test-user@KRBTEST.COM" created. 11:18:19.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy 11:18:19.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 11:18:19.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "joe-interloper@KRBTEST.COM" created. 11:18:19.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal slave/admin@KRBTEST.COM with password. 11:18:19.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/test-user.keytab. 11:18:19.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/test-user.keytab. 11:18:19.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/test-user.keytab. 11:18:19.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal test-user with kvno 1, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/test-user.keytab. 11:18:19.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:19 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:19.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Password for test-admin@KRBTEST.COM: 11:18:19.871 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.3.207.254@KRBTEST.COM; defaulting to no policy 11:18:19.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:19.874 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.3.207.254@KRBTEST.COM" created. 11:18:19.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:19.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.254.keytab. 11:18:19.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.254.keytab. 11:18:19.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.254 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.254.keytab. 11:18:19.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.254 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.254.keytab. 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.889240 3903 external_mini_cluster.cc:933] Running /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/wal 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/data 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.207.254:37007 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.207.254 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/logs 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/data/info.pb 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.254.keytab 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.207.254 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=1 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authz_token_validity_seconds=1 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 11:18:19.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.3.207.254:37007,127.3.207.253:33137,127.3.207.252:34077 with env {KRB5CCNAME=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 11:18:19.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 11:18:19.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:19.942355 3922 flags.cc:404] Enabled unsafe flag: --rpc_inject_invalid_authn_token_ratio=0.5 11:18:19.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:19.942553 3922 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 11:18:19.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:19.942567 3922 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.3.207.254 11:18:19.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:19.942605 3922 flags.cc:404] Enabled unsafe flag: --never_fsync=true 11:18:19.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:19.943645 3922 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=1 11:18:19.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:19.943668 3922 flags.cc:404] Enabled experimental flag: --authz_token_validity_seconds=1 11:18:19.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:19.943677 3922 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 11:18:19.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:19.943693 3922 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 11:18:19.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:19.943701 3922 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 11:18:19.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:19.943712 3922 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 11:18:19.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:19.943727 3922 flags.cc:404] Enabled experimental flag: --principal=kudu/127.3.207.254 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.946120 3922 master_main.cc:73] Master server non-default flags: 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/data 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/wal 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=1 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authz_token_validity_seconds=1 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.3.207.254:37007,127.3.207.253:33137,127.3.207.252:34077 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.207.254:37007 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.254.keytab 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.207.254 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/data/info.pb 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.207.254 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --heap_profile_path=/tmp/kudu-master.3922 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:19.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 11:18:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 11:18:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/logs 11:18:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 11:18:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 11:18:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-cdh6.x-SNAPSHOT 11:18:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5327ca21851e41b4ad88c45c6cffc929a77ae8f4 11:18:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type DEBUG 11:18:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 06 Apr 2019 03:41:25 PST on quasar-mflytl-1.vpc.cloudera.com 11:18:19.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 304881 11:18:19.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.952168 3922 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 46354us 11:18:19.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:19 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549499, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:19.953 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.953610 3922 init.cc:359] Logged in from keytab as kudu/127.3.207.254@KRBTEST.COM (short username kudu) 11:18:19.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.954545 3922 fs_manager.cc:263] Metadata directory not provided 11:18:19.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.954586 3922 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 11:18:19.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.954843 3922 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 11:18:19.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.954876 3922 server_base.cc:434] Attempting to create new FS layout instead 11:18:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.956724 3922 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/data/instance: 11:18:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "ccce7dcecc6c474c87bd8f6710734ba7" 11:18:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:19 on dist-test-slave-dist-test-slave-6ztd" 11:18:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.956939 3922 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/wal/instance: 11:18:19.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "ccce7dcecc6c474c87bd8f6710734ba7" 11:18:19.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:19 on dist-test-slave-dist-test-slave-6ztd" 11:18:19.961 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.961685 3922 fs_manager.cc:503] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s 11:18:19.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.962278 3922 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 11:18:19.962 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.962709 3922 file_cache.cc:466] Constructed file cache lbm with capacity 419430 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.965456 3922 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.965545 3922 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/data,/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/wal 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "ccce7dcecc6c474c87bd8f6710734ba7" 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:19 on dist-test-slave-dist-test-slave-6ztd" 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:19.965647 3922 fs_report.cc:352] FS layout report 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/wal 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/wal 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/data/data 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 11:18:19.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 11:18:19.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 11:18:19.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 11:18:19.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 11:18:19.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 11:18:19.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 11:18:19.966 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 11:18:20.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.000138 3922 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 11:18:20.004 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.004853 3922 rpc_server.cc:205] RPC server started. Bound to: 127.3.207.254:37007 11:18:20.005 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.005532 3922 webserver.cc:308] Webserver started at http://127.3.207.254:35327/ using document root and password file 11:18:20.006 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.006006 3922 server_base.cc:608] Dumped server information to /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-0/data/info.pb 11:18:20.008 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.008597 3982 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:20.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.011765 3982 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } has no permanent_uuid. Determining permanent_uuid... 11:18:20.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.013510 3903 external_mini_cluster.cc:995] Started /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master as pid 3922 11:18:20.018 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549499, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.254@KRBTEST.COM for kudu/127.3.207.254@KRBTEST.COM 11:18:20.020 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.3.207.253@KRBTEST.COM; defaulting to no policy 11:18:20.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:20.022 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.3.207.253@KRBTEST.COM" created. 11:18:20.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.024039 3982 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } has no permanent_uuid. Determining permanent_uuid... 11:18:20.024 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.024684 3982 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.207.253:33137: Network error: Client connection negotiation failed: client connection to 127.3.207.253:33137: connect: Connection refused (error 111) 11:18:20.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:20.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.253.keytab. 11:18:20.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.253.keytab. 11:18:20.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.253 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.253.keytab. 11:18:20.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.253 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.253.keytab. 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.039263 3903 external_mini_cluster.cc:933] Running /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/wal 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/data 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.207.253:33137 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.207.253 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/logs 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/data/info.pb 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.253.keytab 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.207.253 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:20.039 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=1 11:18:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authz_token_validity_seconds=1 11:18:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 11:18:20.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.3.207.254:37007,127.3.207.253:33137,127.3.207.252:34077 with env {KRB5CCNAME=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 11:18:20.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.043891 3982 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } attempt: 1 11:18:20.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.044617 3982 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.207.253:33137: Network error: Client connection negotiation failed: client connection to 127.3.207.253:33137: connect: Connection refused (error 111) 11:18:20.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 11:18:20.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.092376 3987 flags.cc:404] Enabled unsafe flag: --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.092700 3987 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 11:18:20.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.092718 3987 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.3.207.253 11:18:20.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.092761 3987 flags.cc:404] Enabled unsafe flag: --never_fsync=true 11:18:20.092 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.092820 3982 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } attempt: 2 11:18:20.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.093602 3982 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.207.253:33137: Network error: Client connection negotiation failed: client connection to 127.3.207.253:33137: connect: Connection refused (error 111) 11:18:20.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.093799 3987 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=1 11:18:20.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.093822 3987 flags.cc:404] Enabled experimental flag: --authz_token_validity_seconds=1 11:18:20.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.093828 3987 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 11:18:20.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.093840 3987 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 11:18:20.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.093847 3987 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 11:18:20.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.093854 3987 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 11:18:20.093 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.093865 3987 flags.cc:404] Enabled experimental flag: --principal=kudu/127.3.207.253 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.096220 3987 master_main.cc:73] Master server non-default flags: 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/data 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/wal 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=1 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authz_token_validity_seconds=1 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.3.207.254:37007,127.3.207.253:33137,127.3.207.252:34077 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.207.253:33137 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.253.keytab 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.207.253 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/data/info.pb 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.207.253 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --heap_profile_path=/tmp/kudu-master.3987 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/logs 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 11:18:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-cdh6.x-SNAPSHOT 11:18:20.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5327ca21851e41b4ad88c45c6cffc929a77ae8f4 11:18:20.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type DEBUG 11:18:20.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 06 Apr 2019 03:41:25 PST on quasar-mflytl-1.vpc.cloudera.com 11:18:20.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 304881 11:18:20.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.101742 3987 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 46854us 11:18:20.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:20.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.103082 3987 init.cc:359] Logged in from keytab as kudu/127.3.207.253@KRBTEST.COM (short username kudu) 11:18:20.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.104135 3987 fs_manager.cc:263] Metadata directory not provided 11:18:20.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.104174 3987 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 11:18:20.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.104354 3987 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 11:18:20.104 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.104373 3987 server_base.cc:434] Attempting to create new FS layout instead 11:18:20.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.106251 3987 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/data/instance: 11:18:20.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "70ecc9ae00ff4004b27a2b261141d0f4" 11:18:20.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.106469 3987 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/wal/instance: 11:18:20.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "70ecc9ae00ff4004b27a2b261141d0f4" 11:18:20.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.111176 3987 fs_manager.cc:503] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s 11:18:20.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.111742 3987 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 11:18:20.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.112123 3987 file_cache.cc:466] Constructed file cache lbm with capacity 419430 11:18:20.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.114786 3987 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 11:18:20.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.114861 3987 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/data,/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/wal 11:18:20.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "70ecc9ae00ff4004b27a2b261141d0f4" 11:18:20.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.114926 3987 fs_report.cc:352] FS layout report 11:18:20.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 11:18:20.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/wal 11:18:20.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/wal 11:18:20.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/data/data 11:18:20.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 11:18:20.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 11:18:20.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 11:18:20.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 11:18:20.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 11:18:20.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 11:18:20.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 11:18:20.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 11:18:20.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 11:18:20.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 11:18:20.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.175940 3987 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 11:18:20.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.179683 3987 rpc_server.cc:205] RPC server started. Bound to: 127.3.207.253:33137 11:18:20.180 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.180398 3987 webserver.cc:308] Webserver started at http://127.3.207.253:42975/ using document root and password file 11:18:20.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.181979 3987 server_base.cc:608] Dumped server information to /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-1/data/info.pb 11:18:20.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.184389 4047 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:20.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.184805 3903 external_mini_cluster.cc:995] Started /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master as pid 3987 11:18:20.187 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.187659 4047 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } has no permanent_uuid. Determining permanent_uuid... 11:18:20.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.3.207.252@KRBTEST.COM; defaulting to no policy 11:18:20.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:20.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.3.207.252@KRBTEST.COM" created. 11:18:20.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.253@KRBTEST.COM for kudu/127.3.207.254@KRBTEST.COM 11:18:20.200 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.200697 4047 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } has no permanent_uuid. Determining permanent_uuid... 11:18:20.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.204748 3982 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } attempt: 3 11:18:20.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.253@KRBTEST.COM for kudu/127.3.207.253@KRBTEST.COM 11:18:20.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549499, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.254@KRBTEST.COM for kudu/127.3.207.253@KRBTEST.COM 11:18:20.210 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.210453 4047 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } has no permanent_uuid. Determining permanent_uuid... 11:18:20.211 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.211050 4047 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.207.252:34077: Network error: Client connection negotiation failed: client connection to 127.3.207.252:34077: connect: Connection refused (error 111) 11:18:20.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.212083 3982 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } has no permanent_uuid. Determining permanent_uuid... 11:18:20.212 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.212651 3982 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.207.252:34077: Network error: Client connection negotiation failed: client connection to 127.3.207.252:34077: connect: Connection refused (error 111) 11:18:20.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:20.214 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.252.keytab. 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.252.keytab. 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.252 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.252.keytab. 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.252 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.252.keytab. 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.215513 3903 external_mini_cluster.cc:933] Running /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/wal 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/data 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=localhost 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.207.252:34077 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.207.252 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/logs 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/data/info.pb 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 11:18:20.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 11:18:20.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 11:18:20.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.252.keytab 11:18:20.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.207.252 11:18:20.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:20.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:20.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:20.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=1 11:18:20.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authz_token_validity_seconds=1 11:18:20.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 11:18:20.216 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.3.207.254:37007,127.3.207.253:33137,127.3.207.252:34077 with env {KRB5CCNAME=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 11:18:20.230 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.230286 4047 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } attempt: 1 11:18:20.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.231106 4047 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.207.252:34077: Network error: Client connection negotiation failed: client connection to 127.3.207.252:34077: connect: Connection refused (error 111) 11:18:20.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 11:18:20.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.268083 4053 flags.cc:404] Enabled unsafe flag: --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.268275 4053 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 11:18:20.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.268285 4053 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.3.207.252 11:18:20.268 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.268312 4053 flags.cc:404] Enabled unsafe flag: --never_fsync=true 11:18:20.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.269299 4053 flags.cc:404] Enabled experimental flag: --authn_token_validity_seconds=1 11:18:20.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.269320 4053 flags.cc:404] Enabled experimental flag: --authz_token_validity_seconds=1 11:18:20.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.269326 4053 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 11:18:20.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.269338 4053 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 11:18:20.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.269345 4053 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 11:18:20.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.269353 4053 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true 11:18:20.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.269363 4053 flags.cc:404] Enabled experimental flag: --principal=kudu/127.3.207.252 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.271621 4053 master_main.cc:73] Master server non-default flags: 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/data 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/wal 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authn_token_validity_seconds=1 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --authz_token_validity_seconds=1 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_ca_key_size=1024 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --master_addresses=127.3.207.254:37007,127.3.207.253:33137,127.3.207.252:34077 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tsk_num_rsa_bits=512 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.207.252:34077 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_reuseport=true 11:18:20.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.252.keytab 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.207.252 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/data/info.pb 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.207.252 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --heap_profile_path=/tmp/kudu-master.4053 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/logs 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Master server version: 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-cdh6.x-SNAPSHOT 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5327ca21851e41b4ad88c45c6cffc929a77ae8f4 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type DEBUG 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 06 Apr 2019 03:41:25 PST on quasar-mflytl-1.vpc.cloudera.com 11:18:20.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 304881 11:18:20.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.276829 3982 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } attempt: 1 11:18:20.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.277122 4053 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 46854us 11:18:20.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.277664 3982 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.207.252:34077: Network error: Client connection negotiation failed: client connection to 127.3.207.252:34077: connect: Connection refused (error 111) 11:18:20.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:20.278 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.278553 4053 init.cc:359] Logged in from keytab as kudu/127.3.207.252@KRBTEST.COM (short username kudu) 11:18:20.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.279287 4047 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } attempt: 2 11:18:20.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.279521 4053 fs_manager.cc:263] Metadata directory not provided 11:18:20.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.279563 4053 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 11:18:20.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.279789 4053 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 11:18:20.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.279814 4053 server_base.cc:434] Attempting to create new FS layout instead 11:18:20.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.279917 4047 consensus_peers.cc:599] Error getting permanent uuid from config peer 127.3.207.252:34077: Network error: Client connection negotiation failed: client connection to 127.3.207.252:34077: connect: Connection refused (error 111) 11:18:20.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.281638 4053 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/data/instance: 11:18:20.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "aa508c067da34f5288753b0f65ef7312" 11:18:20.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.281 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.281915 4053 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/wal/instance: 11:18:20.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "aa508c067da34f5288753b0f65ef7312" 11:18:20.282 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.286703 4053 fs_manager.cc:503] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s 11:18:20.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.287331 4053 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 11:18:20.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.287714 4053 file_cache.cc:466] Constructed file cache lbm with capacity 419430 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.290423 4053 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.290498 4053 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/data,/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/wal 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "aa508c067da34f5288753b0f65ef7312" 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.290560 4053 fs_report.cc:352] FS layout report 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/wal 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/wal 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/data/data 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 11:18:20.290 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 11:18:20.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.326167 4053 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 11:18:20.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.331455 4053 rpc_server.cc:205] RPC server started. Bound to: 127.3.207.252:34077 11:18:20.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.332152 4053 webserver.cc:308] Webserver started at http://127.3.207.252:34521/ using document root and password file 11:18:20.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.332659 4053 server_base.cc:608] Dumped server information to /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/master-2/data/info.pb 11:18:20.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.335045 4113 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:20.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.338465 4113 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } has no permanent_uuid. Determining permanent_uuid... 11:18:20.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.339808 3903 external_mini_cluster.cc:995] Started /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master as pid 4053 11:18:20.345 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.252@KRBTEST.COM for kudu/127.3.207.254@KRBTEST.COM 11:18:20.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.3.207.193@KRBTEST.COM; defaulting to no policy 11:18:20.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:20.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.3.207.193@KRBTEST.COM" created. 11:18:20.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.350679 4113 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } has no permanent_uuid. Determining permanent_uuid... 11:18:20.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.252@KRBTEST.COM for kudu/127.3.207.253@KRBTEST.COM 11:18:20.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.356799 3982 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } attempt: 2 11:18:20.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.357278 4113 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } has no permanent_uuid. Determining permanent_uuid... 11:18:20.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549499, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.254@KRBTEST.COM for kudu/127.3.207.252@KRBTEST.COM 11:18:20.361 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.252@KRBTEST.COM for kudu/127.3.207.252@KRBTEST.COM 11:18:20.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:20.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.193.keytab. 11:18:20.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.193.keytab. 11:18:20.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.193 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.193.keytab. 11:18:20.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.193 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.193.keytab. 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.369236 4113 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312: Bootstrap starting. 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.369236 3982 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7: Bootstrap starting. 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.369451 3903 external_mini_cluster.cc:933] Running /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/wal 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/data 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.207.193:0 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.3.207.193 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.207.193 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.3.207.254:37007,127.3.207.253:33137,127.3.207.252:34077 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:20.369 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/logs 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/data/info.pb 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.193.keytab 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.207.193 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_enforce_access_control=true 11:18:20.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_inject_invalid_authz_token_ratio=0.5 with env {KRB5CCNAME=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 11:18:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.371398 4113 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312: No blocks or log segments found. Creating new log. 11:18:20.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.371402 3982 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7: No blocks or log segments found. Creating new log. 11:18:20.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.372197 4113 log.cc:525] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312: Log is configured to *not* fsync() on all Append() calls 11:18:20.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.372197 3982 log.cc:525] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7: Log is configured to *not* fsync() on all Append() calls 11:18:20.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.373538 4113 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312: No bootstrap required, opened a new log 11:18:20.373 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.373755 3982 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7: No bootstrap required, opened a new log 11:18:20.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.376466 3982 raft_consensus.cc:340] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } 11:18:20.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.376471 4113 raft_consensus.cc:340] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } 11:18:20.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.376538 4113 raft_consensus.cc:366] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:20.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.376538 3982 raft_consensus.cc:366] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:20.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.376575 3982 raft_consensus.cc:705] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccce7dcecc6c474c87bd8f6710734ba7, State: Initialized, Role: FOLLOWER 11:18:20.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.376577 4113 raft_consensus.cc:705] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa508c067da34f5288753b0f65ef7312, State: Initialized, Role: FOLLOWER 11:18:20.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.377104 3982 consensus_queue.cc:229] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } 11:18:20.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.377104 4113 consensus_queue.cc:229] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } 11:18:20.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.377427 4113 sys_catalog.cc:337] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.377427 3982 sys_catalog.cc:337] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.377460 3982 sys_catalog.cc:340] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [sys.catalog]: This master's current role is: FOLLOWER 11:18:20.377 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.377460 4113 sys_catalog.cc:340] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [sys.catalog]: This master's current role is: FOLLOWER 11:18:20.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.378381 3982 sys_catalog.cc:424] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [sys.catalog]: configured and running, proceeding with master startup. 11:18:20.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.378381 4113 sys_catalog.cc:424] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [sys.catalog]: configured and running, proceeding with master startup. 11:18:20.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.380758 4123 sys_catalog.cc:337] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.380 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.380786 4123 sys_catalog.cc:340] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [sys.catalog]: This master's current role is: FOLLOWER 11:18:20.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.380998 4122 sys_catalog.cc:337] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.381024 4122 sys_catalog.cc:340] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [sys.catalog]: This master's current role is: FOLLOWER 11:18:20.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.383352 4124 catalog_manager.cc:1113] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312: acquiring CA information for follower catalog manager: Not found: root CA entry not found 11:18:20.383 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.383386 4124 catalog_manager.cc:594] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 11:18:20.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.385510 4125 catalog_manager.cc:1113] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7: acquiring CA information for follower catalog manager: Not found: root CA entry not found 11:18:20.385 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.385546 4125 catalog_manager.cc:594] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 11:18:20.391 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.391095 4047 consensus_peers.cc:609] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } attempt: 3 11:18:20.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.253@KRBTEST.COM for kudu/127.3.207.252@KRBTEST.COM 11:18:20.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.405458 4047 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4: Bootstrap starting. 11:18:20.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.407737 4047 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4: No blocks or log segments found. Creating new log. 11:18:20.408 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.408529 4047 log.cc:525] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4: Log is configured to *not* fsync() on all Append() calls 11:18:20.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.413553 4047 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4: No bootstrap required, opened a new log 11:18:20.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.416757 4047 raft_consensus.cc:340] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } 11:18:20.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.416997 4047 raft_consensus.cc:366] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:20.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.417163 4047 raft_consensus.cc:705] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ecc9ae00ff4004b27a2b261141d0f4, State: Initialized, Role: FOLLOWER 11:18:20.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.418113 4047 consensus_queue.cc:229] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } 11:18:20.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.418606 4047 sys_catalog.cc:337] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.418769 4047 sys_catalog.cc:340] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [sys.catalog]: This master's current role is: FOLLOWER 11:18:20.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.419870 4047 sys_catalog.cc:424] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [sys.catalog]: configured and running, proceeding with master startup. 11:18:20.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.421864 4129 sys_catalog.cc:337] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.421 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.421891 4129 sys_catalog.cc:340] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [sys.catalog]: This master's current role is: FOLLOWER 11:18:20.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.427110 4130 catalog_manager.cc:1113] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4: acquiring CA information for follower catalog manager: Not found: root CA entry not found 11:18:20.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.427263 4130 catalog_manager.cc:594] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry 11:18:20.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 11:18:20.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.427527 4119 flags.cc:404] Enabled unsafe flag: --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.427939 4119 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 11:18:20.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.428055 4119 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.3.207.193 11:18:20.428 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.428194 4119 flags.cc:404] Enabled unsafe flag: --never_fsync=true 11:18:20.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.429476 4119 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 11:18:20.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.429646 4119 flags.cc:404] Enabled experimental flag: --principal=kudu/127.3.207.193 11:18:20.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.429777 4119 flags.cc:404] Enabled experimental flag: --tserver_enforce_access_control=true 11:18:20.429 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.429888 4119 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.207.193 11:18:20.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.430899 4003 tablet.cc:1786] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4: Can't schedule compaction. Clean time has not been advanced past its initial value. 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.433526 4119 tablet_server_main.cc:78] Tablet server non-default flags: 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/data 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/wal 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.207.193:0 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.193.keytab 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.207.193 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:20.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/data/info.pb 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.207.193 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.3.207.254:37007,127.3.207.253:33137,127.3.207.252:34077 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_enforce_access_control=true 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_inject_invalid_authz_token_ratio=0.5 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --heap_profile_path=/tmp/kudu-tserver.4119 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.3.207.193 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/logs 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-cdh6.x-SNAPSHOT 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5327ca21851e41b4ad88c45c6cffc929a77ae8f4 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type DEBUG 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 06 Apr 2019 03:41:25 PST on quasar-mflytl-1.vpc.cloudera.com 11:18:20.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 304881 11:18:20.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.443281 4119 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 46854us 11:18:20.444 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:20.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.445329 4119 init.cc:359] Logged in from keytab as kudu/127.3.207.193@KRBTEST.COM (short username kudu) 11:18:20.446 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.446506 4119 fs_manager.cc:263] Metadata directory not provided 11:18:20.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.447055 4119 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 11:18:20.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.447407 4119 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 11:18:20.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.447515 4119 server_base.cc:434] Attempting to create new FS layout instead 11:18:20.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.449904 4119 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/data/instance: 11:18:20.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "ded7f95fdab24558bd552201f0175add" 11:18:20.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.450414 4119 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/wal/instance: 11:18:20.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "ded7f95fdab24558bd552201f0175add" 11:18:20.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.456800 4119 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s 11:18:20.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.457713 4119 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 11:18:20.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.458353 4119 file_cache.cc:466] Constructed file cache lbm with capacity 419430 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.462132 4119 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.462368 4119 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/data,/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/wal 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "ded7f95fdab24558bd552201f0175add" 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.462647 4119 fs_report.cc:352] FS layout report 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/wal 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/wal 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/data/data 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 11:18:20.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 11:18:20.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 11:18:20.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 11:18:20.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 11:18:20.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 11:18:20.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.503410 3938 tablet.cc:1786] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7: Can't schedule compaction. Clean time has not been advanced past its initial value. 11:18:20.509 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.508958 4119 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 11:18:20.510 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.510489 4119 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) 11:18:20.514 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.514698 4119 ts_tablet_manager.cc:376] Registered 0 tablets 11:18:20.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.519806 4122 raft_consensus.cc:472] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 11:18:20.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.520015 4122 raft_consensus.cc:494] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } 11:18:20.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.521790 4122 leader_election.cc:296] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70ecc9ae00ff4004b27a2b261141d0f4 (127.3.207.253:33137), aa508c067da34f5288753b0f65ef7312 (127.3.207.252:34077) 11:18:20.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.523080 4024 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" is_pre_election: true 11:18:20.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.523320 4024 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccce7dcecc6c474c87bd8f6710734ba7 in term 0. 11:18:20.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.523849 4083 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa508c067da34f5288753b0f65ef7312" is_pre_election: true 11:18:20.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.524058 4083 raft_consensus.cc:2346] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccce7dcecc6c474c87bd8f6710734ba7 in term 0. 11:18:20.525 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.524574 3932 leader_election.cc:310] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa508c067da34f5288753b0f65ef7312, ccce7dcecc6c474c87bd8f6710734ba7; no voters: 11:18:20.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.526046 4122 raft_consensus.cc:2640] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [term 0 FOLLOWER]: Leader pre-election won for term 1 11:18:20.527 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.526093 4122 raft_consensus.cc:472] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 11:18:20.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.526114 4122 raft_consensus.cc:2886] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [term 0 FOLLOWER]: Advancing to term 1 11:18:20.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.527266 4122 raft_consensus.cc:494] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } 11:18:20.528 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.527549 4122 leader_election.cc:296] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [CANDIDATE]: Term 1 election: Requested vote from peers 70ecc9ae00ff4004b27a2b261141d0f4 (127.3.207.253:33137), aa508c067da34f5288753b0f65ef7312 (127.3.207.252:34077) 11:18:20.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.528565 4083 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa508c067da34f5288753b0f65ef7312" 11:18:20.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.528741 4024 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" 11:18:20.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.528810 4024 raft_consensus.cc:2886] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [term 0 FOLLOWER]: Advancing to term 1 11:18:20.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.529894 4024 raft_consensus.cc:2346] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccce7dcecc6c474c87bd8f6710734ba7 in term 1. 11:18:20.531 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.530230 3931 leader_election.cc:310] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70ecc9ae00ff4004b27a2b261141d0f4, ccce7dcecc6c474c87bd8f6710734ba7; no voters: 11:18:20.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.530405 4083 raft_consensus.cc:2886] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [term 0 FOLLOWER]: Advancing to term 1 11:18:20.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.531438 4083 raft_consensus.cc:2346] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccce7dcecc6c474c87bd8f6710734ba7 in term 1. 11:18:20.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.532167 4122 raft_consensus.cc:2640] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [term 1 FOLLOWER]: Leader election won for term 1 11:18:20.532 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.532239 4122 raft_consensus.cc:667] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [term 1 LEADER]: Becoming Leader. State: Replica: ccce7dcecc6c474c87bd8f6710734ba7, State: Running, Role: LEADER 11:18:20.533 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.532899 4122 consensus_queue.cc:206] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } 11:18:20.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.534938 4122 sys_catalog.cc:337] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccce7dcecc6c474c87bd8f6710734ba7. Latest consensus state: current_term: 1 leader_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.535109 4122 sys_catalog.cc:340] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [sys.catalog]: This master's current role is: LEADER 11:18:20.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.535961 4222 catalog_manager.cc:1055] Loading table and tablet metadata into memory... 11:18:20.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.538062 4222 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... 11:18:20.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.543190 4119 rpc_server.cc:205] RPC server started. Bound to: 127.3.207.193:39213 11:18:20.544 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.544224 4119 webserver.cc:308] Webserver started at http://127.3.207.193:34599/ using document root and password file 11:18:20.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.548842 4119 server_base.cc:608] Dumped server information to /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-0/data/info.pb 11:18:20.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.554509 3903 external_mini_cluster.cc:995] Started /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver as pid 4119 11:18:20.558 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.193@KRBTEST.COM for kudu/127.3.207.253@KRBTEST.COM 11:18:20.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.193@KRBTEST.COM for kudu/127.3.207.254@KRBTEST.COM 11:18:20.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.193@KRBTEST.COM for kudu/127.3.207.252@KRBTEST.COM 11:18:20.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.3.207.194@KRBTEST.COM; defaulting to no policy 11:18:20.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.566932 4256 heartbeater.cc:345] Connected to a master server at 127.3.207.254:37007 11:18:20.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.566936 4255 heartbeater.cc:345] Connected to a master server at 127.3.207.253:33137 11:18:20.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.567214 4256 heartbeater.cc:426] Registering TS with master... 11:18:20.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.567215 4255 heartbeater.cc:426] Registering TS with master... 11:18:20.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.567607 4256 heartbeater.cc:475] Master 127.3.207.254:37007 requested a full tablet report, sending... 11:18:20.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.567608 4255 heartbeater.cc:475] Master 127.3.207.253:33137 requested a full tablet report, sending... 11:18:20.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:20.568 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.3.207.194@KRBTEST.COM" created. 11:18:20.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.568909 3948 ts_manager.cc:159] Registered new tserver with Master: ded7f95fdab24558bd552201f0175add (127.3.207.193:39213) 11:18:20.569 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.568979 4014 ts_manager.cc:159] Registered new tserver with Master: ded7f95fdab24558bd552201f0175add (127.3.207.193:39213) 11:18:20.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.570749 4254 heartbeater.cc:345] Connected to a master server at 127.3.207.252:34077 11:18:20.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.570787 4254 heartbeater.cc:426] Registering TS with master... 11:18:20.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.570940 4254 heartbeater.cc:475] Master 127.3.207.252:34077 requested a full tablet report, sending... 11:18:20.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.571789 4083 raft_consensus.cc:1181] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [term 1 FOLLOWER]: Refusing update from remote peer ccce7dcecc6c474c87bd8f6710734ba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 11:18:20.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.571831 4024 raft_consensus.cc:1181] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [term 1 FOLLOWER]: Refusing update from remote peer ccce7dcecc6c474c87bd8f6710734ba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) 11:18:20.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.572479 4220 consensus_queue.cc:984] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:20.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.572563 4122 consensus_queue.cc:984] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:20.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.573390 4077 ts_manager.cc:159] Registered new tserver with Master: ded7f95fdab24558bd552201f0175add (127.3.207.193:39213) 11:18:20.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.576766 4129 sys_catalog.cc:337] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccce7dcecc6c474c87bd8f6710734ba7. Latest consensus state: current_term: 1 leader_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.576793 4129 sys_catalog.cc:340] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [sys.catalog]: This master's current role is: FOLLOWER 11:18:20.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.577150 4069 tablet.cc:1786] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312: Can't schedule compaction. Clean time has not been advanced past its initial value. 11:18:20.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.578014 4123 sys_catalog.cc:337] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccce7dcecc6c474c87bd8f6710734ba7. Latest consensus state: current_term: 1 leader_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.578063 4123 sys_catalog.cc:340] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [sys.catalog]: This master's current role is: FOLLOWER 11:18:20.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.581189 4122 sys_catalog.cc:337] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.581280 4122 sys_catalog.cc:340] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [sys.catalog]: This master's current role is: LEADER 11:18:20.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.581569 4129 sys_catalog.cc:337] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.581598 4129 sys_catalog.cc:340] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4 [sys.catalog]: This master's current role is: FOLLOWER 11:18:20.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.582680 4222 catalog_manager.cc:950] Generated new certificate authority record 11:18:20.584 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.583963 4222 catalog_manager.cc:1075] Loading token signing keys... 11:18:20.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.584915 4122 sys_catalog.cc:337] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.585 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.584939 4122 sys_catalog.cc:340] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7 [sys.catalog]: This master's current role is: LEADER 11:18:20.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.586012 4123 sys_catalog.cc:337] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccce7dcecc6c474c87bd8f6710734ba7" member_type: VOTER last_known_addr { host: "127.3.207.254" port: 37007 } } peers { permanent_uuid: "70ecc9ae00ff4004b27a2b261141d0f4" member_type: VOTER last_known_addr { host: "127.3.207.253" port: 33137 } } peers { permanent_uuid: "aa508c067da34f5288753b0f65ef7312" member_type: VOTER last_known_addr { host: "127.3.207.252" port: 34077 } } } 11:18:20.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.586048 4123 sys_catalog.cc:340] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312 [sys.catalog]: This master's current role is: FOLLOWER 11:18:20.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:20.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.194.keytab. 11:18:20.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.194.keytab. 11:18:20.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.194 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.194.keytab. 11:18:20.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.194 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.194.keytab. 11:18:20.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.588800 3903 external_mini_cluster.cc:933] Running /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver 11:18:20.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver 11:18:20.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/wal 11:18:20.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/data 11:18:20.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.207.194:0 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.3.207.194 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.207.194 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.3.207.254:37007,127.3.207.253:33137,127.3.207.252:34077 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/logs 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/data/info.pb 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.194.keytab 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.207.194 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_enforce_access_control=true 11:18:20.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_inject_invalid_authz_token_ratio=0.5 with env {KRB5CCNAME=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 11:18:20.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.606855 4222 catalog_manager.cc:4280] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7: Generated new TSK 0 11:18:20.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 11:18:20.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.639562 4272 flags.cc:404] Enabled unsafe flag: --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.639750 4272 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 11:18:20.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.639765 4272 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.3.207.194 11:18:20.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.639811 4272 flags.cc:404] Enabled unsafe flag: --never_fsync=true 11:18:20.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.640779 4272 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 11:18:20.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.640818 4272 flags.cc:404] Enabled experimental flag: --principal=kudu/127.3.207.194 11:18:20.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.640857 4272 flags.cc:404] Enabled experimental flag: --tserver_enforce_access_control=true 11:18:20.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.640889 4272 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.207.194 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.642995 4272 tablet_server_main.cc:78] Tablet server non-default flags: 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/data 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/wal 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.207.194:0 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.194.keytab 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.207.194 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/data/info.pb 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.207.194 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.3.207.254:37007,127.3.207.253:33137,127.3.207.252:34077 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_enforce_access_control=true 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_inject_invalid_authz_token_ratio=0.5 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --heap_profile_path=/tmp/kudu-tserver.4272 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.3.207.194 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/logs 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 11:18:20.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-cdh6.x-SNAPSHOT 11:18:20.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5327ca21851e41b4ad88c45c6cffc929a77ae8f4 11:18:20.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type DEBUG 11:18:20.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 06 Apr 2019 03:41:25 PST on quasar-mflytl-1.vpc.cloudera.com 11:18:20.644 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 304881 11:18:20.649 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.649210 4272 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 46854us 11:18:20.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:20.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.650545 4272 init.cc:359] Logged in from keytab as kudu/127.3.207.194@KRBTEST.COM (short username kudu) 11:18:20.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.651404 4272 fs_manager.cc:263] Metadata directory not provided 11:18:20.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.651443 4272 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 11:18:20.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.651665 4272 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 11:18:20.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.651690 4272 server_base.cc:434] Attempting to create new FS layout instead 11:18:20.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.653479 4272 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/data/instance: 11:18:20.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "de226502f41746669a7be49b5bb6d0ea" 11:18:20.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.653762 4272 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/wal/instance: 11:18:20.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "de226502f41746669a7be49b5bb6d0ea" 11:18:20.653 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.658383 4272 fs_manager.cc:503] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s 11:18:20.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.658970 4272 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 11:18:20.659 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.659401 4272 file_cache.cc:466] Constructed file cache lbm with capacity 419430 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.661972 4272 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.662088 4272 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/data,/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/wal 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "de226502f41746669a7be49b5bb6d0ea" 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.662220 4272 fs_report.cc:352] FS layout report 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/wal 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/wal 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/data/data 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 11:18:20.662 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 11:18:20.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.696341 4272 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 11:18:20.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.697557 4272 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) 11:18:20.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.697584 4272 ts_tablet_manager.cc:376] Registered 0 tablets 11:18:20.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.703466 4272 rpc_server.cc:205] RPC server started. Bound to: 127.3.207.194:37233 11:18:20.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.704210 4272 webserver.cc:308] Webserver started at http://127.3.207.194:44741/ using document root and password file 11:18:20.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.704717 4272 server_base.cc:608] Dumped server information to /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/data/info.pb 11:18:20.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.706575 3903 external_mini_cluster.cc:995] Started /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver as pid 4272 11:18:20.713 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: no policy specified for kudu/127.3.207.195@KRBTEST.COM; defaulting to no policy 11:18:20.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:20.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Principal "kudu/127.3.207.195@KRBTEST.COM" created. 11:18:20.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.194@KRBTEST.COM for kudu/127.3.207.252@KRBTEST.COM 11:18:20.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.194@KRBTEST.COM for kudu/127.3.207.253@KRBTEST.COM 11:18:20.718 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.194@KRBTEST.COM for kudu/127.3.207.254@KRBTEST.COM 11:18:20.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.722301 4392 heartbeater.cc:345] Connected to a master server at 127.3.207.252:34077 11:18:20.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.722556 4392 heartbeater.cc:426] Registering TS with master... 11:18:20.722 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.722926 4392 heartbeater.cc:475] Master 127.3.207.252:34077 requested a full tablet report, sending... 11:18:20.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.723153 4394 heartbeater.cc:345] Connected to a master server at 127.3.207.254:37007 11:18:20.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.723198 4394 heartbeater.cc:426] Registering TS with master... 11:18:20.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.723312 4394 heartbeater.cc:475] Master 127.3.207.254:37007 requested a full tablet report, sending... 11:18:20.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.722309 4393 heartbeater.cc:345] Connected to a master server at 127.3.207.253:33137 11:18:20.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.723750 4393 heartbeater.cc:426] Registering TS with master... 11:18:20.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.723851 4077 ts_manager.cc:159] Registered new tserver with Master: de226502f41746669a7be49b5bb6d0ea (127.3.207.194:37233) 11:18:20.723 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.723940 4393 heartbeater.cc:475] Master 127.3.207.253:33137 requested a full tablet report, sending... 11:18:20.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.723953 3948 ts_manager.cc:159] Registered new tserver with Master: de226502f41746669a7be49b5bb6d0ea (127.3.207.194:37233) 11:18:20.724 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.724563 4014 ts_manager.cc:159] Registered new tserver with Master: de226502f41746669a7be49b5bb6d0ea (127.3.207.194:37233) 11:18:20.725 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.725536 3948 master_service.cc:271] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.207.194@KRBTEST.COM'} at 127.3.207.194:50221 11:18:20.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Authenticating as principal test-admin/admin@KRBTEST.COM with password. 11:18:20.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.195.keytab. 11:18:20.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.195.keytab. 11:18:20.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.195 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.195.keytab. 11:18:20.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Entry for principal kudu/127.3.207.195 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.195.keytab. 11:18:20.735 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.735947 3903 external_mini_cluster.cc:933] Running /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/wal 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/data 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --block_manager=log 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.207.195:0 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.3.207.195 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.207.195 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.3.207.254:37007,127.3.207.253:33137,127.3.207.252:34077 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/logs 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/data/info.pb 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.195.keytab 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.207.195 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_enforce_access_control=true 11:18:20.736 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_inject_invalid_authz_token_ratio=0.5 with env {KRB5CCNAME=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} 11:18:20.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) WARNING: Logging before InitGoogleLogging() is written to STDERR 11:18:20.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.781581 4401 flags.cc:404] Enabled unsafe flag: --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.781766 4401 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true 11:18:20.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.781780 4401 flags.cc:404] Enabled unsafe flag: --principal=kudu/127.3.207.195 11:18:20.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.781827 4401 flags.cc:404] Enabled unsafe flag: --never_fsync=true 11:18:20.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.782824 4401 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 11:18:20.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.782860 4401 flags.cc:404] Enabled experimental flag: --principal=kudu/127.3.207.195 11:18:20.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.782902 4401 flags.cc:404] Enabled experimental flag: --tserver_enforce_access_control=true 11:18:20.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.782938 4401 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.207.195 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.785110 4401 tablet_server_main.cc:78] Tablet server non-default flags: 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_data_dirs=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/data 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --fs_wal_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/wal 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_inject_invalid_authn_token_ratio=0.5 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --ipki_server_key_size=1024 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_bind_addresses=127.3.207.195:0 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_server_allow_ephemeral_ports=true 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --keytab_file=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kudu_127.3.207.195.keytab 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --principal=kudu/127.3.207.195 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --rpc_authentication=required 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --superuser_acl=test-admin 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --user_acl=test-user 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --metrics_log_interval_ms=1000 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_format=pb 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --server_dump_info_path=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/data/info.pb 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_interface=127.3.207.195 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --webserver_port=0 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_master_addrs=127.3.207.254:37007,127.3.207.253:33137,127.3.207.252:34077 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_enforce_access_control=true 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --tserver_inject_invalid_authz_token_ratio=0.5 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --never_fsync=true 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --heap_profile_path=/tmp/kudu-tserver.4401 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --redact=none 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_experimental_flags=true 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --unlock_unsafe_flags=true 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --enable_minidumps=false 11:18:20.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --local_ip_for_outbound_sockets=127.3.207.195 11:18:20.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --log_dir=/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/logs 11:18:20.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logbuflevel=-1 11:18:20.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) --logtostderr=true 11:18:20.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Tablet server version: 11:18:20.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) kudu 1.10.0-cdh6.x-SNAPSHOT 11:18:20.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) revision 5327ca21851e41b4ad88c45c6cffc929a77ae8f4 11:18:20.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build type DEBUG 11:18:20.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) built by jenkins at 06 Apr 2019 03:41:25 PST on quasar-mflytl-1.vpc.cloudera.com 11:18:20.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) build id 304881 11:18:20.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.791270 4401 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 46854us 11:18:20.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM 11:18:20.792 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.792678 4401 init.cc:359] Logged in from keytab as kudu/127.3.207.195@KRBTEST.COM (short username kudu) 11:18:20.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.793524 4401 fs_manager.cc:263] Metadata directory not provided 11:18:20.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.793565 4401 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory 11:18:20.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.793803 4401 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file 11:18:20.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.793830 4401 server_base.cc:434] Attempting to create new FS layout instead 11:18:20.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.795580 4401 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/data/instance: 11:18:20.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" 11:18:20.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.795876 4401 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/wal/instance: 11:18:20.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" 11:18:20.796 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.800 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.800848 4401 fs_manager.cc:503] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s 11:18:20.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.801441 4401 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go 11:18:20.801 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.801821 4401 file_cache.cc:466] Constructed file cache lbm with capacity 419430 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.804322 4401 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.804404 4401 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/data,/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/wal 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) format_stamp: "Formatted at 2019-04-06 11:18:20 on dist-test-slave-dist-test-slave-6ztd" 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.804471 4401 fs_report.cc:352] FS layout report 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) -------------------- 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) wal directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/wal 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) metadata directory: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/wal 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 1 data directories: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/data/data 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live blocks: 0 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes: 0 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total live bytes (after alignment): 0 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total number of LBM containers: 0 (0 full) 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for missing blocks 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Did not check for orphaned blocks 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM containers with extra space: 0 (0 repaired) 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total full LBM container extra space in bytes: 0 (0 repaired) 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total incomplete LBM containers: 0 (0 repaired) 11:18:20.804 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Total LBM partial records: 0 (0 repaired) 11:18:20.830 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.830358 4401 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go 11:18:20.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.831270 4401 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) 11:18:20.831 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.831295 4401 ts_tablet_manager.cc:376] Registered 0 tablets 11:18:20.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.836992 4401 rpc_server.cc:205] RPC server started. Bound to: 127.3.207.195:46293 11:18:20.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.837759 4401 webserver.cc:308] Webserver started at http://127.3.207.195:33077/ using document root and password file 11:18:20.838 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.838253 4401 server_base.cc:608] Dumped server information to /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/data/info.pb 11:18:20.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.839932 3903 external_mini_cluster.cc:995] Started /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver as pid 4401 11:18:20.846 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.195@KRBTEST.COM for kudu/127.3.207.254@KRBTEST.COM 11:18:20.847 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.195@KRBTEST.COM for kudu/127.3.207.253@KRBTEST.COM 11:18:20.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549500, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.195@KRBTEST.COM for kudu/127.3.207.252@KRBTEST.COM 11:18:20.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.852358 4522 heartbeater.cc:345] Connected to a master server at 127.3.207.253:33137 11:18:20.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.852357 4523 heartbeater.cc:345] Connected to a master server at 127.3.207.254:37007 11:18:20.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.852658 4523 heartbeater.cc:426] Registering TS with master... 11:18:20.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.207.252@KRBTEST.COM 11:18:20.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.852982 4523 heartbeater.cc:475] Master 127.3.207.254:37007 requested a full tablet report, sending... 11:18:20.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.853154 4521 heartbeater.cc:345] Connected to a master server at 127.3.207.252:34077 11:18:20.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.853193 4521 heartbeater.cc:426] Registering TS with master... 11:18:20.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.853305 4522 heartbeater.cc:426] Registering TS with master... 11:18:20.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.853314 4521 heartbeater.cc:475] Master 127.3.207.252:34077 requested a full tablet report, sending... 11:18:20.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.853554 4522 heartbeater.cc:475] Master 127.3.207.253:33137 requested a full tablet report, sending... 11:18:20.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.853782 3948 ts_manager.cc:159] Registered new tserver with Master: b66c7f5abb6c4bb2ae977d0de8b441f7 (127.3.207.195:46293) 11:18:20.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.854300 4014 ts_manager.cc:159] Registered new tserver with Master: b66c7f5abb6c4bb2ae977d0de8b441f7 (127.3.207.195:46293) 11:18:20.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.854656 3948 master_service.cc:271] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.207.195@KRBTEST.COM'} at 127.3.207.195:51093 11:18:20.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.855588 4077 ts_manager.cc:159] Registered new tserver with Master: b66c7f5abb6c4bb2ae977d0de8b441f7 (127.3.207.195:46293) 11:18:20.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.207.253@KRBTEST.COM 11:18:20.867 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.207.254@KRBTEST.COM 11:18:20.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.870666 3903 external_mini_cluster.cc:623] 3 TS(s) registered with all masters 11:18:20.870 [DEBUG - main] (MiniKuduCluster.java:165) Response: 11:18:20.871 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_kdc_env_vars { } 11:18:20.872 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_kdc_env_vars { env_vars { key: "KRB5_CONFIG" value: "/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5.conf" } env_vars { key: "KRB5CCNAME" value: "/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5cc" } env_vars { key: "KUDU_ENABLE_KRB5_REALM_FIX" value: "yes" } env_vars { key: "KRB5_KDC_PROFILE" value: "/tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/kdc.conf" } } 11:18:20.873 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_masters { } 11:18:20.873 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_masters { masters { id { type: MASTER index: 0 } bound_rpc_address { host: "127.3.207.254" port: 37007 } } masters { id { type: MASTER index: 1 } bound_rpc_address { host: "127.3.207.253" port: 33137 } } masters { id { type: MASTER index: 2 } bound_rpc_address { host: "127.3.207.252" port: 34077 } } } 11:18:20.874 [DEBUG - main] (MiniKuduCluster.java:155) Request: get_tservers { } 11:18:20.874 [DEBUG - main] (MiniKuduCluster.java:165) Response: get_tservers { tservers { id { type: TSERVER index: 0 } bound_rpc_address { host: "127.3.207.193" port: 39213 } } tservers { id { type: TSERVER index: 1 } bound_rpc_address { host: "127.3.207.194" port: 37233 } } tservers { id { type: TSERVER index: 2 } bound_rpc_address { host: "127.3.207.195" port: 46293 } } } 11:18:20.874 [INFO - main] (KuduTestHarness.java:141) Creating a new Kudu client... 11:18:20.881 [DEBUG - main] (SecurityUtil.java:106) Using ticketCache: /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/krb5kdc/krb5cc 11:18:20.882 [DEBUG - main] (SecurityUtil.java:120) Logged in as subject: test-admin@KRBTEST.COM 11:18:20.899 [DEBUG - New I/O worker #10] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.207.252 11:18:20.900 [DEBUG - New I/O worker #11] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.207.254 11:18:20.900 [DEBUG - New I/O worker #12] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.207.253 11:18:20.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.207.253@KRBTEST.COM 11:18:20.923 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.207.252@KRBTEST.COM 11:18:20.928 [DEBUG - New I/O worker #12] (Negotiator.java:764) Authenticated connection [id: 0x0353bda4, /127.0.0.1:39998 => /127.3.207.253:33137] using SASL/GSSAPI 11:18:20.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.207.254@KRBTEST.COM 11:18:20.935 [DEBUG - New I/O worker #10] (Negotiator.java:764) Authenticated connection [id: 0x4e3d14be, /127.0.0.1:39850 => /127.3.207.252:34077] using SASL/GSSAPI 11:18:20.940 [DEBUG - New I/O worker #11] (Negotiator.java:764) Authenticated connection [id: 0xe0b2fbf7, /127.0.0.1:53768 => /127.3.207.254:37007] using SASL/GSSAPI 11:18:20.943 [DEBUG - New I/O worker #11] (AsyncKuduClient.java:2206) Learned about tablet Kudu Master for table 'Kudu Master' with partition [, ) 11:18:20.943 [DEBUG - New I/O worker #11] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=CreateTable, tablet=null, attempt=1, TimeoutTracker(timeout=50000, elapsed=60), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.207.252:34077, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.207.254:37007, [1ms] Sub RPC ConnectToMaster: sending RPC to server master-127.3.207.253:33137, [46ms] Sub RPC ConnectToMaster: received response from server master-127.3.207.253:33137: OK, [57ms] Sub RPC ConnectToMaster: received response from server master-127.3.207.252:34077: OK, [59ms] Sub RPC ConnectToMaster: received response from server master-127.3.207.254:37007: OK, deferred=null) after lookup 11:18:20.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.945919 3948 catalog_manager.cc:1415] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53768: 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "TestAuthTokenReacquire-table" 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) schema { 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: true 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column1_i" 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column2_i" 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: INT32 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column3_s" 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: STRING 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: true 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) encoding: DICT_ENCODING 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) compression: LZ4 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 4096 11:18:20.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "column4_b" 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) type: BOOL 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_key: false 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) is_nullable: false 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) cfile_block_size: 0 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) partition_schema { 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) range_schema { 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) columns { 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) name: "key" 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) } 11:18:20.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:20.946882 3948 catalog_manager.cc:1574] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAuthTokenReacquire-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. 11:18:20.957 [DEBUG - New I/O worker #11] (AsyncKuduClient.java:791) Opened table 342026b4b67742bd9505404eed3879b7 11:18:20.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Apr 06 11:18:20 dist-test-slave-dist-test-slave-6ztd krb5kdc[3911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1554549499, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.207.254@KRBTEST.COM for kudu/127.3.207.193@KRBTEST.COM 11:18:20.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.966612 4444 tablet_service.cc:1006] Processing CreateTablet for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 (table=TestAuthTokenReacquire-table [id=342026b4b67742bd9505404eed3879b7]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:20.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.966619 4326 tablet_service.cc:1006] Processing CreateTablet for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 (table=TestAuthTokenReacquire-table [id=342026b4b67742bd9505404eed3879b7]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:20.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.967074 4444 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:20.968 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.967079 4326 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:20.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.972788 4175 tablet_service.cc:1006] Processing CreateTablet for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 (table=TestAuthTokenReacquire-table [id=342026b4b67742bd9505404eed3879b7]), partition=RANGE (key) PARTITION UNBOUNDED 11:18:20.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.972975 4545 tablet_bootstrap.cc:439] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea: Bootstrap starting. 11:18:20.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.973209 4175 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1. 1 dirs total, 0 dirs full, 0 dirs failed 11:18:20.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.974442 4545 tablet_bootstrap.cc:590] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea: No blocks or log segments found. Creating new log. 11:18:20.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.975008 4545 log.cc:525] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea: Log is configured to *not* fsync() on all Append() calls 11:18:20.976 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.975425 4546 tablet_bootstrap.cc:439] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7: Bootstrap starting. 11:18:20.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.980876 4548 tablet_bootstrap.cc:439] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add: Bootstrap starting. 11:18:20.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.982121 4548 tablet_bootstrap.cc:590] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add: No blocks or log segments found. Creating new log. 11:18:20.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.982658 4548 log.cc:525] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add: Log is configured to *not* fsync() on all Append() calls 11:18:20.982 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.982873 4546 tablet_bootstrap.cc:590] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7: No blocks or log segments found. Creating new log. 11:18:20.983 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.983402 4546 log.cc:525] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7: Log is configured to *not* fsync() on all Append() calls 11:18:20.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.984377 4546 tablet_bootstrap.cc:439] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7: No bootstrap required, opened a new log 11:18:20.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.984521 4548 tablet_bootstrap.cc:439] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add: No bootstrap required, opened a new log 11:18:20.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.984555 4546 ts_tablet_manager.cc:1053] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s 11:18:20.984 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.984719 4548 ts_tablet_manager.cc:1053] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s 11:18:20.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.985142 4545 tablet_bootstrap.cc:439] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea: No bootstrap required, opened a new log 11:18:20.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.985306 4545 ts_tablet_manager.cc:1053] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s 11:18:20.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.986845 4548 raft_consensus.cc:340] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de226502f41746669a7be49b5bb6d0ea" member_type: VOTER last_known_addr { host: "127.3.207.194" port: 37233 } } peers { permanent_uuid: "ded7f95fdab24558bd552201f0175add" member_type: VOTER last_known_addr { host: "127.3.207.193" port: 39213 } } peers { permanent_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" member_type: VOTER last_known_addr { host: "127.3.207.195" port: 46293 } } 11:18:20.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.986914 4548 raft_consensus.cc:366] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:20.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.986955 4548 raft_consensus.cc:705] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded7f95fdab24558bd552201f0175add, State: Initialized, Role: FOLLOWER 11:18:20.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.987336 4548 consensus_queue.cc:229] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de226502f41746669a7be49b5bb6d0ea" member_type: VOTER last_known_addr { host: "127.3.207.194" port: 37233 } } peers { permanent_uuid: "ded7f95fdab24558bd552201f0175add" member_type: VOTER last_known_addr { host: "127.3.207.193" port: 39213 } } peers { permanent_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" member_type: VOTER last_known_addr { host: "127.3.207.195" port: 46293 } } 11:18:20.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.987885 4545 raft_consensus.cc:340] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de226502f41746669a7be49b5bb6d0ea" member_type: VOTER last_known_addr { host: "127.3.207.194" port: 37233 } } peers { permanent_uuid: "ded7f95fdab24558bd552201f0175add" member_type: VOTER last_known_addr { host: "127.3.207.193" port: 39213 } } peers { permanent_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" member_type: VOTER last_known_addr { host: "127.3.207.195" port: 46293 } } 11:18:20.988 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.988606 4548 ts_tablet_manager.cc:1081] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s 11:18:20.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.990401 4546 raft_consensus.cc:340] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de226502f41746669a7be49b5bb6d0ea" member_type: VOTER last_known_addr { host: "127.3.207.194" port: 37233 } } peers { permanent_uuid: "ded7f95fdab24558bd552201f0175add" member_type: VOTER last_known_addr { host: "127.3.207.193" port: 39213 } } peers { permanent_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" member_type: VOTER last_known_addr { host: "127.3.207.195" port: 46293 } } 11:18:20.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.990455 4546 raft_consensus.cc:366] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:20.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.990475 4546 raft_consensus.cc:705] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66c7f5abb6c4bb2ae977d0de8b441f7, State: Initialized, Role: FOLLOWER 11:18:20.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.990665 3948 master_service.cc:271] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.207.193@KRBTEST.COM'} at 127.3.207.193:49423 11:18:20.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.990773 4546 consensus_queue.cc:229] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de226502f41746669a7be49b5bb6d0ea" member_type: VOTER last_known_addr { host: "127.3.207.194" port: 37233 } } peers { permanent_uuid: "ded7f95fdab24558bd552201f0175add" member_type: VOTER last_known_addr { host: "127.3.207.193" port: 39213 } } peers { permanent_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" member_type: VOTER last_known_addr { host: "127.3.207.195" port: 46293 } } 11:18:20.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.991124 4545 raft_consensus.cc:366] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely 11:18:20.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.991312 4545 raft_consensus.cc:705] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de226502f41746669a7be49b5bb6d0ea, State: Initialized, Role: FOLLOWER 11:18:20.992 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.992148 4256 heartbeater.cc:467] Master 127.3.207.254:37007 was elected leader, sending a full tablet report... 11:18:20.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.992512 4523 heartbeater.cc:467] Master 127.3.207.254:37007 was elected leader, sending a full tablet report... 11:18:20.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.993474 4545 consensus_queue.cc:229] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de226502f41746669a7be49b5bb6d0ea" member_type: VOTER last_known_addr { host: "127.3.207.194" port: 37233 } } peers { permanent_uuid: "ded7f95fdab24558bd552201f0175add" member_type: VOTER last_known_addr { host: "127.3.207.193" port: 39213 } } peers { permanent_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" member_type: VOTER last_known_addr { host: "127.3.207.195" port: 46293 } } 11:18:20.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.993785 4546 ts_tablet_manager.cc:1081] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s 11:18:20.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.994354 4394 heartbeater.cc:467] Master 127.3.207.254:37007 was elected leader, sending a full tablet report... 11:18:20.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:20.994918 4545 ts_tablet_manager.cc:1081] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s 11:18:21.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.035962 4552 raft_consensus.cc:472] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) 11:18:21.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.036167 4552 raft_consensus.cc:494] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de226502f41746669a7be49b5bb6d0ea" member_type: VOTER last_known_addr { host: "127.3.207.194" port: 37233 } } peers { permanent_uuid: "ded7f95fdab24558bd552201f0175add" member_type: VOTER last_known_addr { host: "127.3.207.193" port: 39213 } } peers { permanent_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" member_type: VOTER last_known_addr { host: "127.3.207.195" port: 46293 } } 11:18:21.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.037088 4552 leader_election.cc:296] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de226502f41746669a7be49b5bb6d0ea (127.3.207.194:37233), ded7f95fdab24558bd552201f0175add (127.3.207.193:39213) 11:18:21.040 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.040884 4349 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "853aa44fa6304a0bbcd3e7cfe7f989e1" candidate_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de226502f41746669a7be49b5bb6d0ea" is_pre_election: true 11:18:21.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.041095 4349 raft_consensus.cc:2346] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b66c7f5abb6c4bb2ae977d0de8b441f7 in term 0. 11:18:21.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.041492 4208 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "853aa44fa6304a0bbcd3e7cfe7f989e1" candidate_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded7f95fdab24558bd552201f0175add" is_pre_election: true 11:18:21.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.041564 4412 leader_election.cc:310] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c7f5abb6c4bb2ae977d0de8b441f7, de226502f41746669a7be49b5bb6d0ea; no voters: 11:18:21.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.041676 4208 raft_consensus.cc:2346] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b66c7f5abb6c4bb2ae977d0de8b441f7 in term 0. 11:18:21.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.041844 4552 raft_consensus.cc:2640] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 0 FOLLOWER]: Leader pre-election won for term 1 11:18:21.041 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.041905 4552 raft_consensus.cc:472] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) 11:18:21.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.041927 4552 raft_consensus.cc:2886] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 0 FOLLOWER]: Advancing to term 1 11:18:21.042 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.042814 4552 raft_consensus.cc:494] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de226502f41746669a7be49b5bb6d0ea" member_type: VOTER last_known_addr { host: "127.3.207.194" port: 37233 } } peers { permanent_uuid: "ded7f95fdab24558bd552201f0175add" member_type: VOTER last_known_addr { host: "127.3.207.193" port: 39213 } } peers { permanent_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" member_type: VOTER last_known_addr { host: "127.3.207.195" port: 46293 } } 11:18:21.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.043089 4552 leader_election.cc:296] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [CANDIDATE]: Term 1 election: Requested vote from peers de226502f41746669a7be49b5bb6d0ea (127.3.207.194:37233), ded7f95fdab24558bd552201f0175add (127.3.207.193:39213) 11:18:21.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.043424 4349 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "853aa44fa6304a0bbcd3e7cfe7f989e1" candidate_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de226502f41746669a7be49b5bb6d0ea" 11:18:21.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.043520 4349 raft_consensus.cc:2886] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea [term 0 FOLLOWER]: Advancing to term 1 11:18:21.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.043552 4208 tablet_service.cc:1276] Received RequestConsensusVote() RPC: tablet_id: "853aa44fa6304a0bbcd3e7cfe7f989e1" candidate_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded7f95fdab24558bd552201f0175add" 11:18:21.043 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.043598 4208 raft_consensus.cc:2886] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add [term 0 FOLLOWER]: Advancing to term 1 11:18:21.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.044394 4208 raft_consensus.cc:2346] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c7f5abb6c4bb2ae977d0de8b441f7 in term 1. 11:18:21.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.044430 4349 raft_consensus.cc:2346] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b66c7f5abb6c4bb2ae977d0de8b441f7 in term 1. 11:18:21.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.044749 4409 leader_election.cc:310] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b66c7f5abb6c4bb2ae977d0de8b441f7, ded7f95fdab24558bd552201f0175add; no voters: 11:18:21.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.044906 4552 raft_consensus.cc:2640] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 1 FOLLOWER]: Leader election won for term 1 11:18:21.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.045111 4552 raft_consensus.cc:667] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 1 LEADER]: Becoming Leader. State: Replica: b66c7f5abb6c4bb2ae977d0de8b441f7, State: Running, Role: LEADER 11:18:21.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.045512 4552 consensus_queue.cc:206] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de226502f41746669a7be49b5bb6d0ea" member_type: VOTER last_known_addr { host: "127.3.207.194" port: 37233 } } peers { permanent_uuid: "ded7f95fdab24558bd552201f0175add" member_type: VOTER last_known_addr { host: "127.3.207.193" port: 39213 } } peers { permanent_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" member_type: VOTER last_known_addr { host: "127.3.207.195" port: 46293 } } 11:18:21.048 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.048465 3943 catalog_manager.cc:3976] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 reported cstate change: term changed from 0 to 1, leader changed from to b66c7f5abb6c4bb2ae977d0de8b441f7 (127.3.207.195). New cstate: current_term: 1 leader_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de226502f41746669a7be49b5bb6d0ea" member_type: VOTER last_known_addr { host: "127.3.207.194" port: 37233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ded7f95fdab24558bd552201f0175add" member_type: VOTER last_known_addr { host: "127.3.207.193" port: 39213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b66c7f5abb6c4bb2ae977d0de8b441f7" member_type: VOTER last_known_addr { host: "127.3.207.195" port: 46293 } health_report { overall_health: HEALTHY } } } 11:18:21.052 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:21.051666 4257 tablet.cc:1786] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add: Can't schedule compaction. Clean time has not been advanced past its initial value. 11:18:21.085 [DEBUG - New I/O worker #12] (Connection.java:691) [peer master-127.3.207.253:33137(127.3.207.253:33137)] cleaning up while in state READY due to: connection disconnected 11:18:21.086 [DEBUG - New I/O worker #11] (Connection.java:691) [peer master-127.3.207.254:37007(127.3.207.254:37007)] cleaning up while in state READY due to: connection disconnected 11:18:21.085 [DEBUG - New I/O worker #10] (Connection.java:691) [peer master-127.3.207.252:34077(127.3.207.252:34077)] cleaning up while in state READY due to: connection disconnected 11:18:21.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:21.089563 4524 tablet.cc:1786] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7: Can't schedule compaction. Clean time has not been advanced past its initial value. 11:18:21.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:21.206455 4395 tablet.cc:1786] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea: Can't schedule compaction. Clean time has not been advanced past its initial value. 11:18:21.386 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.386250 4124 catalog_manager.cc:1111] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312: acquiring CA information for follower catalog manager: success 11:18:21.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.387285 4124 catalog_manager.cc:1139] T 00000000000000000000000000000000 P aa508c067da34f5288753b0f65ef7312: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 11:18:21.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.430469 4130 catalog_manager.cc:1111] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4: acquiring CA information for follower catalog manager: success 11:18:21.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.431627 4130 catalog_manager.cc:1139] T 00000000000000000000000000000000 P 70ecc9ae00ff4004b27a2b261141d0f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 11:18:21.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.608837 4349 raft_consensus.cc:1181] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea [term 1 FOLLOWER]: Refusing update from remote peer b66c7f5abb6c4bb2ae977d0de8b441f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:21.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.609580 4556 consensus_queue.cc:984] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de226502f41746669a7be49b5bb6d0ea" member_type: VOTER last_known_addr { host: "127.3.207.194" port: 37233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:21.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.614873 4208 raft_consensus.cc:1181] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add [term 1 FOLLOWER]: Refusing update from remote peer b66c7f5abb6c4bb2ae977d0de8b441f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) 11:18:21.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:21.615476 4559 consensus_queue.cc:984] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ded7f95fdab24558bd552201f0175add" member_type: VOTER last_known_addr { host: "127.3.207.193" port: 39213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s 11:18:23.110 [DEBUG - New I/O worker #13] (Negotiator.java:291) peer /127.3.207.254:37007 sent connection negotiation error: Not authorized: token expired 11:18:23.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:23.110100 4568 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53778: token expired 11:18:23.110 [DEBUG - New I/O worker #13] (Connection.java:691) [peer master-127.3.207.254:37007(127.3.207.254:37007)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:23.113 [DEBUG - New I/O worker #14] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.207.252 11:18:23.114 [DEBUG - New I/O worker #15] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.207.254 11:18:23.117 [DEBUG - New I/O worker #16] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.207.253 11:18:23.123 [DEBUG - New I/O worker #14] (Negotiator.java:764) Authenticated connection [id: 0x0cd9cf37, /127.0.0.1:39864 => /127.3.207.252:34077] using SASL/GSSAPI 11:18:23.126 [DEBUG - New I/O worker #16] (Negotiator.java:764) Authenticated connection [id: 0x8e027809, /127.0.0.1:40012 => /127.3.207.253:33137] using SASL/GSSAPI 11:18:23.133 [DEBUG - New I/O worker #15] (Negotiator.java:764) Authenticated connection [id: 0x44320d5f, /127.0.0.1:53782 => /127.3.207.254:37007] using SASL/GSSAPI 11:18:23.135 [INFO - New I/O worker #15] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:23.137 [DEBUG - New I/O worker #15] (AsyncKuduClient.java:2206) Learned about tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 for table 'TestAuthTokenReacquire-table' with partition [, ) 11:18:23.146 [DEBUG - New I/O worker #15] (TableLocationsCache.java:188) Discovered table locations: [Tablet{lowerBoundPartitionKey=0x, upperBoundPartitionKey=0x, ttl=299991, tablet=853aa44fa6304a0bbcd3e7cfe7f989e1@[b66c7f5abb6c4bb2ae977d0de8b441f7(127.3.207.195:46293)[L],de226502f41746669a7be49b5bb6d0ea(127.3.207.194:37233),ded7f95fdab24558bd552201f0175add(127.3.207.193:39213)]}] 11:18:23.147 [DEBUG - New I/O worker #15] (AsyncKuduClient.java:1316) Retrying sending RPC KuduRpc(method=Write, tablet=null, attempt=1, TimeoutTracker(timeout=60000, elapsed=51), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.207.254:37007, [13ms] Sub RPC GetTableLocations: waiting for new authn token, [37ms] Sub RPC GetTableLocations: sending RPC to server master-127.3.207.254:37007, [38ms] Sub RPC GetTableLocations: received response from server master-127.3.207.254:37007: OK, deferred=null) after lookup 11:18:23.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:23.163998 4571 fault_injection.cc:44] FAULT INJECTION ENABLED! 11:18:23.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:23.164038 4571 fault_injection.cc:45] THIS SERVER MAY CRASH! 11:18:23.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:23.164402 4571 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43504: invalid token 11:18:23.164 [DEBUG - New I/O worker #17] (Negotiator.java:291) peer /127.3.207.195:46293 sent connection negotiation error: Not authorized: invalid token 11:18:23.165 [DEBUG - New I/O worker #17] (Connection.java:691) [peer b66c7f5abb6c4bb2ae977d0de8b441f7(127.3.207.195:46293)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:23.166 [INFO - New I/O worker #17] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:23.174 [DEBUG - New I/O worker #10] (Negotiator.java:764) Authenticated connection [id: 0x0a1ef0b7, /127.0.0.1:43506 => /127.3.207.195:46293] using TOKEN/null 11:18:23.177 [DEBUG - New I/O worker #10] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:23.179 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:23.181 [DEBUG - New I/O worker #10] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:23.183 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:23.184 [DEBUG - New I/O worker #10] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:23.186 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:25.206 [DEBUG - New I/O worker #10] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:25.208 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:25.209 [DEBUG - New I/O worker #10] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:25.211 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:25.212 [DEBUG - New I/O worker #10] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:25.214 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:25.220 [DEBUG - New I/O worker #15] (Connection.java:691) [peer master-127.3.207.254:37007(127.3.207.254:37007)] cleaning up while in state READY due to: connection disconnected 11:18:25.220 [DEBUG - New I/O worker #14] (Connection.java:691) [peer master-127.3.207.252:34077(127.3.207.252:34077)] cleaning up while in state READY due to: connection disconnected 11:18:25.220 [DEBUG - New I/O worker #16] (Connection.java:691) [peer master-127.3.207.253:33137(127.3.207.253:33137)] cleaning up while in state READY due to: connection disconnected 11:18:25.221 [DEBUG - New I/O worker #10] (Connection.java:691) [peer b66c7f5abb6c4bb2ae977d0de8b441f7(127.3.207.195:46293)] cleaning up while in state READY due to: connection disconnected 11:18:27.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:27.226534 4593 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53790: token expired 11:18:27.226 [DEBUG - New I/O worker #11] (Negotiator.java:291) peer /127.3.207.254:37007 sent connection negotiation error: Not authorized: token expired 11:18:27.227 [DEBUG - New I/O worker #11] (Connection.java:691) [peer master-127.3.207.254:37007(127.3.207.254:37007)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:27.229 [DEBUG - New I/O worker #13] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.207.254 11:18:27.229 [DEBUG - New I/O worker #12] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.207.252 11:18:27.238 [DEBUG - New I/O worker #14] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.207.253 11:18:27.238 [DEBUG - New I/O worker #12] (Negotiator.java:764) Authenticated connection [id: 0x982c8cdc, /127.0.0.1:39876 => /127.3.207.252:34077] using SASL/GSSAPI 11:18:27.238 [DEBUG - New I/O worker #13] (Negotiator.java:764) Authenticated connection [id: 0x9c4dbd0c, /127.0.0.1:53794 => /127.3.207.254:37007] using SASL/GSSAPI 11:18:27.241 [INFO - New I/O worker #13] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:27.244 [DEBUG - New I/O worker #13] (AsyncKuduClient.java:791) Opened table 342026b4b67742bd9505404eed3879b7 11:18:27.254 [DEBUG - New I/O worker #14] (Negotiator.java:764) Authenticated connection [id: 0x1f7f1402, /127.0.0.1:40024 => /127.3.207.253:33137] using SASL/GSSAPI 11:18:27.269 [DEBUG - New I/O worker #15] (Negotiator.java:764) Authenticated connection [id: 0x473945da, /127.0.0.1:43516 => /127.3.207.195:46293] using TOKEN/null 11:18:27.282 [DEBUG - New I/O worker #15] (AsyncKuduScanner.java:1132) AsyncKuduScanner$Response(scannerId = "", data = RowResultIterator for 2 rows, more = false) for scanner KuduScanner(table=TestAuthTokenReacquire-table, tablet=853aa44fa6304a0bbcd3e7cfe7f989e1, scannerId=null, scanRequestTimeout=60000) 11:18:29.284 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:29.286 [DEBUG - New I/O worker #13] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:29.288 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:29.289 [DEBUG - New I/O worker #13] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:29.290 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:29.291 [DEBUG - New I/O worker #13] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:29.293 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:29.294 [DEBUG - New I/O worker #13] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:29.295 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:29.297 [DEBUG - New I/O worker #13] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:29.298 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:29.299 [DEBUG - New I/O worker #13] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:29.300 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:29.302 [DEBUG - New I/O worker #13] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:29.303 [DEBUG - New I/O worker #15] (AuthzTokenCache.java:151) sending RPC to retrieve token for table ID 342026b4b67742bd9505404eed3879b7 11:18:29.304 [DEBUG - New I/O worker #13] (AuthzTokenCache.java:191) retrieved authz token for 342026b4b67742bd9505404eed3879b7 11:18:29.306 [DEBUG - New I/O worker #15] (AsyncKuduScanner.java:1132) AsyncKuduScanner$Response(scannerId = "", data = RowResultIterator for 2 rows, more = false) for scanner KuduScanner(table=TestAuthTokenReacquire-table, tablet=853aa44fa6304a0bbcd3e7cfe7f989e1, scannerId=null, scanRequestTimeout=60000) 11:18:29.307 [DEBUG - New I/O worker #14] (Connection.java:691) [peer master-127.3.207.253:33137(127.3.207.253:33137)] cleaning up while in state READY due to: connection disconnected 11:18:29.307 [DEBUG - New I/O worker #12] (Connection.java:691) [peer master-127.3.207.252:34077(127.3.207.252:34077)] cleaning up while in state READY due to: connection disconnected 11:18:29.307 [DEBUG - New I/O worker #13] (Connection.java:691) [peer master-127.3.207.254:37007(127.3.207.254:37007)] cleaning up while in state READY due to: connection disconnected 11:18:29.308 [DEBUG - New I/O worker #15] (Connection.java:691) [peer b66c7f5abb6c4bb2ae977d0de8b441f7(127.3.207.195:46293)] cleaning up while in state READY due to: connection disconnected 11:18:31.313 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:31.313537 4600 negotiation.cc:320] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53800: token expired 11:18:31.313 [DEBUG - New I/O worker #16] (Negotiator.java:291) peer /127.3.207.254:37007 sent connection negotiation error: Not authorized: token expired 11:18:31.314 [DEBUG - New I/O worker #16] (Connection.java:691) [peer master-127.3.207.254:37007(127.3.207.254:37007)] cleaning up while in state NEGOTIATION_FAILED due to: connection disconnected 11:18:31.316 [DEBUG - New I/O worker #17] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.207.252 11:18:31.317 [DEBUG - New I/O worker #10] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.207.254 11:18:31.320 [DEBUG - New I/O worker #11] (Negotiator.java:460) SASL mechanism GSSAPI chosen for peer 127.3.207.253 11:18:31.324 [DEBUG - New I/O worker #17] (Negotiator.java:764) Authenticated connection [id: 0xe394e711, /127.0.0.1:39886 => /127.3.207.252:34077] using SASL/GSSAPI 11:18:31.324 [DEBUG - New I/O worker #10] (Negotiator.java:764) Authenticated connection [id: 0x6c6ef051, /127.0.0.1:53804 => /127.3.207.254:37007] using SASL/GSSAPI 11:18:31.326 [INFO - New I/O worker #10] (AsyncKuduClient.java:465) connect to master: received a new authn token 11:18:31.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.327692 3943 catalog_manager.cc:1889] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53804: 11:18:31.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) table { table_name: "TestAuthTokenReacquire-table" } 11:18:31.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.333571 3943 catalog_manager.cc:4215] T 00000000000000000000000000000000 P ccce7dcecc6c474c87bd8f6710734ba7: Sending DeleteTablet for 3 replicas of tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 11:18:31.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.333729 3943 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 on de226502f41746669a7be49b5bb6d0ea (127.3.207.194:37233) (Table deleted at 2019-04-06 11:18:31 UTC) 11:18:31.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.333878 3943 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 on ded7f95fdab24558bd552201f0175add (127.3.207.193:39213) (Table deleted at 2019-04-06 11:18:31 UTC) 11:18:31.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.333951 3943 catalog_manager.cc:3378] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 on b66c7f5abb6c4bb2ae977d0de8b441f7 (127.3.207.195:46293) (Table deleted at 2019-04-06 11:18:31 UTC) 11:18:31.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.334842 4444 tablet_service.cc:1047] Processing DeleteTablet for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:31 UTC) from {username='kudu', principal='kudu/127.3.207.254@KRBTEST.COM'} at 127.0.0.1:43492 11:18:31.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.335016 4326 tablet_service.cc:1047] Processing DeleteTablet for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:31 UTC) from {username='kudu', principal='kudu/127.3.207.254@KRBTEST.COM'} at 127.0.0.1:41738 11:18:31.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.335965 4175 tablet_service.cc:1047] Processing DeleteTablet for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2019-04-06 11:18:31 UTC) from {username='kudu', principal='kudu/127.3.207.254@KRBTEST.COM'} at 127.0.0.1:60084 11:18:31.336 [DEBUG - New I/O worker #10] (Connection.java:691) [peer master-127.3.207.254:37007(127.3.207.254:37007)] cleaning up while in state READY due to: connection disconnected 11:18:31.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.337168 4612 tablet_replica.cc:264] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7: stopping tablet replica 11:18:31.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:31.337883 4602 negotiation.cc:313] Failed RPC negotiation. Trace: 11:18:31.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:31.316075 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40034 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:31.316642 (+ 567us) server_negotiation.cc:184] Beginning negotiation 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:31.316647 (+ 5us) server_negotiation.cc:373] Waiting for connection header 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:31.316662 (+ 15us) server_negotiation.cc:381] Connection header received 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:31.316707 (+ 45us) server_negotiation.cc:337] Received NEGOTIATE NegotiatePB request 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:31.316711 (+ 4us) server_negotiation.cc:420] Received NEGOTIATE request from client 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:31.316778 (+ 67us) server_negotiation.cc:349] Sending NEGOTIATE NegotiatePB response 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:31.316837 (+ 59us) server_negotiation.cc:205] Negotiated authn=SASL 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:31.335231 (+ 18394us) server_negotiation.cc:337] Received TLS_HANDSHAKE NegotiatePB request 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:31.335364 (+ 133us) server_negotiation.cc:349] Sending TLS_HANDSHAKE NegotiatePB response 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) 0406 11:18:31.337330 (+ 1966us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40034: BlockingRecv error: recv got EOF from 127.0.0.1:40034 (error 108) 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":68,"threads_started":1} 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.338060 4615 tablet_replica.cc:264] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add: stopping tablet replica 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.338341 4615 raft_consensus.cc:2143] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.338568 4613 tablet_replica.cc:264] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea: stopping tablet replica 11:18:31.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.338903 4612 raft_consensus.cc:2143] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 1 LEADER]: Raft consensus shutting down. 11:18:31.339 [INFO - New I/O worker #11] (Connection.java:440) [peer master-127.3.207.253:33137(127.3.207.253:33137)] disconnected from peer 11:18:31.339 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.339252 4613 raft_consensus.cc:2143] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea [term 1 FOLLOWER]: Raft consensus shutting down. 11:18:31.339 [DEBUG - New I/O worker #11] (Connection.java:691) [peer master-127.3.207.253:33137(127.3.207.253:33137)] cleaning up while in state NEGOTIATING due to: [peer master-127.3.207.253:33137(127.3.207.253:33137)] disconnected from peer 11:18:31.339 [INFO - New I/O worker #11] (ConnectToCluster.java:361) Unable to connect to master 127.3.207.253:33137: [peer master-127.3.207.253:33137(127.3.207.253:33137)] disconnected from peer 11:18:31.344 [DEBUG - New I/O worker #17] (AsyncKuduClient.java:2373) Releasing all remaining resources 11:18:31.346 [DEBUG - New I/O worker #17] (Connection.java:691) [peer master-127.3.207.252:34077(127.3.207.252:34077)] cleaning up while in state READY due to: connection disconnected 11:18:31.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.340783 4615 raft_consensus.cc:2170] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:31.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.340971 4613 raft_consensus.cc:2170] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:31.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.343107 4615 ts_tablet_manager.cc:1377] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P ded7f95fdab24558bd552201f0175add: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:31.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.344614 4612 raft_consensus.cc:2170] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7 [term 1 FOLLOWER]: Raft consensus is shut down! 11:18:31.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.345417 4612 ts_tablet_manager.cc:1377] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:31.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.346477 4613 ts_tablet_manager.cc:1377] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea: Deleting tablet data with delete state TABLET_DATA_DELETED 11:18:31.348 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.346724 3903 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver with pid 4119 11:18:31.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.349686 4612 ts_tablet_manager.cc:1390] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3 11:18:31.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.349747 4612 log.cc:990] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-2/wal/wals/853aa44fa6304a0bbcd3e7cfe7f989e1 11:18:31.350 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.350019 4612 ts_tablet_manager.cc:1411] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P b66c7f5abb6c4bb2ae977d0de8b441f7: Deleting consensus metadata 11:18:31.351 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.351569 3933 catalog_manager.cc:3356] TS b66c7f5abb6c4bb2ae977d0de8b441f7 (127.3.207.195:46293): tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 (table TestAuthTokenReacquire-table [id=342026b4b67742bd9505404eed3879b7]) successfully deleted 11:18:31.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.353819 4613 ts_tablet_manager.cc:1390] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3 11:18:31.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.353987 4613 log.cc:990] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea: Deleting WAL directory at /tmp/dist-test-taskpieVMq/test-tmp/mini-kudu-cluster4091782174969801631/ts-1/wal/wals/853aa44fa6304a0bbcd3e7cfe7f989e1 11:18:31.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.354400 4613 ts_tablet_manager.cc:1411] T 853aa44fa6304a0bbcd3e7cfe7f989e1 P de226502f41746669a7be49b5bb6d0ea: Deleting consensus metadata 11:18:31.357 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.357661 3933 catalog_manager.cc:3356] TS de226502f41746669a7be49b5bb6d0ea (127.3.207.194:37233): tablet 853aa44fa6304a0bbcd3e7cfe7f989e1 (table TestAuthTokenReacquire-table [id=342026b4b67742bd9505404eed3879b7]) successfully deleted 11:18:31.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) W0406 11:18:31.357755 3930 catalog_manager.cc:3120] TS ded7f95fdab24558bd552201f0175add (127.3.207.193:39213): Delete Tablet RPC failed for tablet 853aa44fa6304a0bbcd3e7cfe7f989e1: Network error: recv got EOF from 127.3.207.193:39213 (error 108) 11:18:31.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.357791 3930 catalog_manager.cc:3156] Scheduling retry of 853aa44fa6304a0bbcd3e7cfe7f989e1 Delete Tablet RPC for TS=ded7f95fdab24558bd552201f0175add with a delay of 28 ms (attempt = 1) 11:18:31.358 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.358450 3903 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver with pid 4272 11:18:31.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.363397 3903 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-tserver with pid 4401 11:18:31.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.368867 3903 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master with pid 3922 11:18:31.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.374379 3903 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master with pid 3987 11:18:31.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:543) I0406 11:18:31.378979 3903 external_mini_cluster.cc:1131] Killing /tmp/dist-test-taskpieVMq/build/debug/bin/kudu-master with pid 4053 Time: 23.362 There was 1 failure: 1) testBasicMasterOperations(org.apache.kudu.client.TestAuthTokenReacquire) java.lang.AssertionError: test failed: unexpected errors at org.junit.Assert.fail(Assert.java:88) at org.apache.kudu.client.TestAuthTokenReacquire.testBasicMasterOperations(TestAuthTokenReacquire.java:153) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:195) at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:212) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77) at org.junit.runner.JUnitCore.main(JUnitCore.java:36) FAILURES!!! Tests run: 2, Failures: 1