[==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from SecurityComponentsFaultsITest [ RUN ] SecurityComponentsFaultsITest.NoKdcOnStart Loading random data Initializing database '/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Sep 27 22:51:57 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): setting up network... Sep 27 22:51:57 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Sep 27 22:51:57 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): skipping unrecognized local address family 17 Sep 27 22:51:57 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): skipping unrecognized local address family 17 krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked Sep 27 22:51:57 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): listening on fd 10: udp fe80::42:acff:fe11:2%eth0.0 Sep 27 22:51:57 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): set up 2 sockets Sep 27 22:51:57 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): commencing operation krb5kdc: starting... WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:51:58.052131 17612 mini_kdc.cc:117] Time spent starting KDC: real 0.357s user 0.002s sys 0.013s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal slave/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal slave/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal slave/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Sep 27 22:51:58 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.17.51.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.17.51.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.17.51.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab. Entry for principal kudu/127.17.51.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab. Entry for principal kudu/127.17.51.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab. Entry for principal kudu/127.17.51.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab. I0927 22:51:58.186281 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.62:11080 --webserver_interface=127.17.51.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab --principal=kudu/127.17.51.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:51:58.355183 17629 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:51:58.355368 17629 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.62 W0927 22:51:58.355429 17629 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:51:58.357484 17629 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:51:58.357554 17629 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0927 22:51:58.357604 17629 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:51:58.357650 17629 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:51:58.357678 17629 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0927 22:51:58.357728 17629 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.62 I0927 22:51:58.361953 17629 master_main.cc:66] Master server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.62:11080 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab --principal=kudu/127.17.51.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data/info.pb --webserver_interface=127.17.51.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:51:58.367233 17629 master_main.cc:73] Initializing master server... I0927 22:51:58.367923 17629 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 50808us Sep 27 22:51:58 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0927 22:51:58.369818 17629 init.cc:339] Logged in from keytab as kudu/127.17.51.62@KRBTEST.COM (short username kudu) I0927 22:51:58.370815 17629 fs_manager.cc:260] Metadata directory not provided I0927 22:51:58.370888 17629 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0927 22:51:58.371125 17629 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0927 22:51:58.371199 17629 server_base.cc:435] Attempting to create new FS layout instead I0927 22:51:58.374328 17629 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data/instance: uuid: "7c96a1ad26d540babd860e4403b869e4" format_stamp: "Formatted at 2018-09-27 22:51:58 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:58.374861 17629 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/wal/instance: uuid: "7c96a1ad26d540babd860e4403b869e4" format_stamp: "Formatted at 2018-09-27 22:51:58 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:58.380358 17629 fs_manager.cc:495] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s I0927 22:51:58.381422 17629 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0927 22:51:58.381698 17629 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0927 22:51:58.384346 17629 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0927 22:51:58.384423 17629 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data,/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/wal uuid: "7c96a1ad26d540babd860e4403b869e4" format_stamp: "Formatted at 2018-09-27 22:51:58 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:58.384589 17629 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0927 22:51:58.413971 17629 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0927 22:51:58.414397 17629 master_main.cc:76] Starting Master server... I0927 22:51:58.414830 17645 process_memory.cc:182] Process hard memory limit is 11.744253 GB I0927 22:51:58.414901 17645 process_memory.cc:184] Process soft memory limit is 9.395403 GB I0927 22:51:58.414970 17645 process_memory.cc:187] Process memory pressure threshold is 7.046552 GB I0927 22:51:58.426021 17629 rpc_server.cc:201] RPC server started. Bound to: 127.17.51.62:11080 I0927 22:51:58.426488 17629 webserver.cc:173] Starting webserver on 127.17.51.62:0 I0927 22:51:58.426544 17629 webserver.cc:184] Document root disabled I0927 22:51:58.427192 17629 webserver.cc:311] Webserver started. Bound to: http://127.17.51.62:39617/ I0927 22:51:58.427901 17629 server_base.cc:607] Dumped server information to /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data/info.pb I0927 22:51:58.430757 17689 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:51:58.437882 17612 external_mini_cluster.cc:871] Started /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master as pid 17629 I0927 22:51:58.437702 17689 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } has no permanent_uuid. Determining permanent_uuid... Sep 27 22:51:58 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.62@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM WARNING: no policy specified for kudu/127.17.51.61@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.17.51.61@KRBTEST.COM" created. I0927 22:51:58.458468 17690 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.440221 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:51:58.443568 (+ 3347us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.443602 (+ 34us) client_negotiation.cc:160] Beginning negotiation 0927 22:51:58.445774 (+ 2172us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:51:58.446545 (+ 771us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:51:58.446551 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:51:58.447110 (+ 559us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:51:58.447207 (+ 97us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:58.447363 (+ 156us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:58.447378 (+ 15us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.447964 (+ 586us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.447967 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:58.448417 (+ 450us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:58.448427 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.449650 (+ 1223us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.449655 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:58.449750 (+ 95us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:58.449943 (+ 193us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:51:58.449945 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:51:58.451753 (+ 1808us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:51:58.454860 (+ 3107us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:58.454867 (+ 7us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:58.454871 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:58.455203 (+ 332us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:58.455921 (+ 718us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:58.455925 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:58.455927 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:58.456001 (+ 74us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:58.456961 (+ 960us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:51:58.456966 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:51:58.457149 (+ 183us) client_negotiation.cc:695] Sending connection context 0927 22:51:58.457419 (+ 270us) client_negotiation.cc:225] Negotiation successful 0927 22:51:58.457622 (+ 203us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":3110,"mutex_wait_us":25,"thread_start_us":2983,"threads_started":1} I0927 22:51:58.458608 17691 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.440220 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40286 0927 22:51:58.444697 (+ 4477us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:58.444702 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:58.444733 (+ 31us) server_negotiation.cc:373] Connection header received 0927 22:51:58.446091 (+ 1358us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:51:58.446098 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:51:58.446338 (+ 240us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:51:58.446500 (+ 162us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:51:58.447547 (+ 1047us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.447830 (+ 283us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.448584 (+ 754us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.449464 (+ 880us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.449582 (+ 118us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:58.452021 (+ 2439us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:51:58.452081 (+ 60us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:51:58.452089 (+ 8us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:51:58.452128 (+ 39us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:51:58.454607 (+ 2479us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:58.455458 (+ 851us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:58.455464 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:58.455466 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:58.455716 (+ 250us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:58.456204 (+ 488us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:58.456207 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:58.456209 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:58.456611 (+ 402us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:51:58.456921 (+ 310us) server_negotiation.cc:901] Waiting for connection context 0927 22:51:58.457665 (+ 744us) server_negotiation.cc:264] Negotiation successful 0927 22:51:58.458196 (+ 531us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":4343,"spinlock_wait_cycles":100736,"thread_start_us":4878,"threads_started":1} I0927 22:51:58.460736 17689 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } has no permanent_uuid. Determining permanent_uuid... I0927 22:51:58.461756 17690 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.461364 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:51:58.461547 (+ 183us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.461619 (+ 72us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":64} W0927 22:51:58.461990 17689 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.61:11081: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.17.51.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.61.keytab. Entry for principal kudu/127.17.51.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.61.keytab. Entry for principal kudu/127.17.51.61 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.61.keytab. Entry for principal kudu/127.17.51.61 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.61.keytab. I0927 22:51:58.478930 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.61:11081 --webserver_interface=127.17.51.61 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.61.keytab --principal=kudu/127.17.51.61 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0927 22:51:58.481264 17689 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } attempt: 1 I0927 22:51:58.482328 17690 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.481917 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:51:58.482137 (+ 220us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.482207 (+ 70us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":77} W0927 22:51:58.482524 17689 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.61:11081: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) I0927 22:51:58.530720 17689 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } attempt: 2 I0927 22:51:58.531868 17690 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.531393 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:51:58.531627 (+ 234us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.531712 (+ 85us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":86} W0927 22:51:58.532078 17689 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.61:11081: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) I0927 22:51:58.643249 17689 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } attempt: 3 WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:51:58.644270 17694 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true I0927 22:51:58.644419 17690 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.643887 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:51:58.644154 (+ 267us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.644252 (+ 98us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":94} W0927 22:51:58.644443 17694 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.61 W0927 22:51:58.644526 17694 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:51:58.644632 17689 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.61:11081: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) W0927 22:51:58.646587 17694 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:51:58.646659 17694 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0927 22:51:58.646687 17694 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:51:58.646729 17694 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:51:58.646759 17694 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0927 22:51:58.646811 17694 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.61 I0927 22:51:58.651023 17694 master_main.cc:66] Master server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/wal --ipki_ca_key_size=1024 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.61:11081 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.61.keytab --principal=kudu/127.17.51.61 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/data/info.pb --webserver_interface=127.17.51.61 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:51:58.655501 17694 master_main.cc:73] Initializing master server... I0927 22:51:58.656085 17694 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 50808us Sep 27 22:51:58 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0927 22:51:58.657795 17694 init.cc:339] Logged in from keytab as kudu/127.17.51.61@KRBTEST.COM (short username kudu) I0927 22:51:58.658582 17694 fs_manager.cc:260] Metadata directory not provided I0927 22:51:58.658648 17694 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0927 22:51:58.658815 17694 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/data/instance: No such file or directory (error 2) I0927 22:51:58.658879 17694 server_base.cc:435] Attempting to create new FS layout instead I0927 22:51:58.661593 17694 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/data/instance: uuid: "2d26965aac2e451c93e86cd8afe389af" format_stamp: "Formatted at 2018-09-27 22:51:58 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:58.662061 17694 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/wal/instance: uuid: "2d26965aac2e451c93e86cd8afe389af" format_stamp: "Formatted at 2018-09-27 22:51:58 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:58.666990 17694 fs_manager.cc:495] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s I0927 22:51:58.667924 17694 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0927 22:51:58.668159 17694 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0927 22:51:58.670955 17694 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0927 22:51:58.671027 17694 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/data,/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/wal uuid: "2d26965aac2e451c93e86cd8afe389af" format_stamp: "Formatted at 2018-09-27 22:51:58 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:58.671159 17694 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0927 22:51:58.720563 17694 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0927 22:51:58.720916 17694 master_main.cc:76] Starting Master server... I0927 22:51:58.721397 17710 process_memory.cc:182] Process hard memory limit is 11.744253 GB I0927 22:51:58.721482 17710 process_memory.cc:184] Process soft memory limit is 9.395403 GB I0927 22:51:58.721534 17710 process_memory.cc:187] Process memory pressure threshold is 7.046552 GB I0927 22:51:58.732780 17694 rpc_server.cc:201] RPC server started. Bound to: 127.17.51.61:11081 I0927 22:51:58.733137 17694 webserver.cc:173] Starting webserver on 127.17.51.61:0 I0927 22:51:58.733197 17694 webserver.cc:184] Document root disabled I0927 22:51:58.733738 17694 webserver.cc:311] Webserver started. Bound to: http://127.17.51.61:37627/ I0927 22:51:58.734253 17694 server_base.cc:607] Dumped server information to /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-1/data/info.pb I0927 22:51:58.737783 17754 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:51:58.738948 17612 external_mini_cluster.cc:871] Started /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master as pid 17694 I0927 22:51:58.745998 17754 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } has no permanent_uuid. Determining permanent_uuid... WARNING: no policy specified for kudu/127.17.51.60@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.17.51.60@KRBTEST.COM" created. Sep 27 22:51:58 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.61@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM I0927 22:51:58.763288 17691 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.748002 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40296 0927 22:51:58.748257 (+ 255us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:58.748261 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:58.748937 (+ 676us) server_negotiation.cc:373] Connection header received 0927 22:51:58.750510 (+ 1573us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:51:58.750515 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:51:58.750623 (+ 108us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:51:58.750744 (+ 121us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:51:58.751761 (+ 1017us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.751970 (+ 209us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.752695 (+ 725us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.753255 (+ 560us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.753342 (+ 87us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:58.757514 (+ 4172us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:51:58.757556 (+ 42us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:51:58.757560 (+ 4us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:51:58.757613 (+ 53us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:51:58.759897 (+ 2284us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:58.760780 (+ 883us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:58.760784 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:58.760786 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:58.761055 (+ 269us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:58.761524 (+ 469us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:58.761528 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:58.761530 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:58.761709 (+ 179us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:51:58.761867 (+ 158us) server_negotiation.cc:901] Waiting for connection context 0927 22:51:58.762620 (+ 753us) server_negotiation.cc:264] Negotiation successful 0927 22:51:58.763017 (+ 397us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":88} I0927 22:51:58.763603 17756 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.748145 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:51:58.748830 (+ 685us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.748858 (+ 28us) client_negotiation.cc:160] Beginning negotiation 0927 22:51:58.750229 (+ 1371us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:51:58.750866 (+ 637us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:51:58.750872 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:51:58.751309 (+ 437us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:51:58.751449 (+ 140us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:58.751585 (+ 136us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:58.751598 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.752106 (+ 508us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.752109 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:58.752586 (+ 477us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:58.752595 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.753417 (+ 822us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.753422 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:58.753535 (+ 113us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:58.753723 (+ 188us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:51:58.753725 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:51:58.757177 (+ 3452us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:51:58.760145 (+ 2968us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:58.760151 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:58.760159 (+ 8us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:58.760566 (+ 407us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:58.761252 (+ 686us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:58.761255 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:58.761257 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:58.761341 (+ 84us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:58.761891 (+ 550us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:51:58.761898 (+ 7us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:51:58.762195 (+ 297us) client_negotiation.cc:695] Sending connection context 0927 22:51:58.762491 (+ 296us) client_negotiation.cc:225] Negotiation successful 0927 22:51:58.762704 (+ 213us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":307,"threads_started":1} I0927 22:51:58.765249 17754 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } has no permanent_uuid. Determining permanent_uuid... Sep 27 22:51:58 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.61@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM I0927 22:51:58.777559 17756 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.765902 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:51:58.766136 (+ 234us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.766151 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:51:58.766232 (+ 81us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:51:58.767258 (+ 1026us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:51:58.767261 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:51:58.767619 (+ 358us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:51:58.767690 (+ 71us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:58.767774 (+ 84us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:58.767782 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.768221 (+ 439us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.768224 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:58.768614 (+ 390us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:58.768623 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.769731 (+ 1108us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.769735 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:58.769880 (+ 145us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:58.770034 (+ 154us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:51:58.770036 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:51:58.771700 (+ 1664us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:51:58.774817 (+ 3117us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:58.774821 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:58.774824 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:58.775146 (+ 322us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:58.775874 (+ 728us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:58.775879 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:58.775881 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:58.775971 (+ 90us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:58.776753 (+ 782us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:51:58.776784 (+ 31us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:51:58.776898 (+ 114us) client_negotiation.cc:695] Sending connection context 0927 22:51:58.777087 (+ 189us) client_negotiation.cc:225] Negotiation successful 0927 22:51:58.777224 (+ 137us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":54} I0927 22:51:58.778162 17758 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.766118 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51092 0927 22:51:58.766893 (+ 775us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:58.766899 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:58.766922 (+ 23us) server_negotiation.cc:373] Connection header received 0927 22:51:58.766966 (+ 44us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:51:58.766971 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:51:58.767126 (+ 155us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:51:58.767239 (+ 113us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:51:58.767896 (+ 657us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.768119 (+ 223us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.768729 (+ 610us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.769570 (+ 841us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.769650 (+ 80us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:58.772050 (+ 2400us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:51:58.772103 (+ 53us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:51:58.772112 (+ 9us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:51:58.772139 (+ 27us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:51:58.774580 (+ 2441us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:58.775371 (+ 791us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:58.775378 (+ 7us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:58.775380 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:58.775619 (+ 239us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:58.776198 (+ 579us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:58.776202 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:58.776204 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:58.776502 (+ 298us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:51:58.776708 (+ 206us) server_negotiation.cc:901] Waiting for connection context 0927 22:51:58.777317 (+ 609us) server_negotiation.cc:264] Negotiation successful 0927 22:51:58.777845 (+ 528us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":665,"thread_start_us":563,"threads_started":1} I0927 22:51:58.779779 17754 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } has no permanent_uuid. Determining permanent_uuid... Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.17.51.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.60.keytab. Entry for principal kudu/127.17.51.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.60.keytab. Entry for principal kudu/127.17.51.60 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.60.keytab. Entry for principal kudu/127.17.51.60 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.60.keytab. I0927 22:51:58.780730 17756 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.780357 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:58.780542 (+ 185us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.780611 (+ 69us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":62} W0927 22:51:58.780930 17754 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:51:58.780951 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.60:11082 --webserver_interface=127.17.51.60 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.60.keytab --principal=kudu/127.17.51.60 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0927 22:51:58.800129 17754 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 1 I0927 22:51:58.800825 17756 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.800518 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:58.800654 (+ 136us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.800709 (+ 55us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":54} W0927 22:51:58.801043 17754 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:51:58.820794 17689 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } attempt: 4 Sep 27 22:51:58 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.62@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM I0927 22:51:58.830652 17690 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.821226 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:51:58.821573 (+ 347us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.821585 (+ 12us) client_negotiation.cc:160] Beginning negotiation 0927 22:51:58.821686 (+ 101us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:51:58.821975 (+ 289us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:51:58.821978 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:51:58.822298 (+ 320us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:51:58.822374 (+ 76us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:58.822447 (+ 73us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:58.822455 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.822828 (+ 373us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.822831 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:58.823170 (+ 339us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:58.823180 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.823924 (+ 744us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.823928 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:58.824008 (+ 80us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:58.824153 (+ 145us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:51:58.824155 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:51:58.825747 (+ 1592us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:51:58.828519 (+ 2772us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:58.828523 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:58.828525 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:58.828799 (+ 274us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:58.829441 (+ 642us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:58.829445 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:58.829446 (+ 1us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:58.829503 (+ 57us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:58.830007 (+ 504us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:51:58.830010 (+ 3us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:51:58.830141 (+ 131us) client_negotiation.cc:695] Sending connection context 0927 22:51:58.830317 (+ 176us) client_negotiation.cc:225] Negotiation successful 0927 22:51:58.830432 (+ 115us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":212} I0927 22:51:58.831013 17758 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.821384 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51098 0927 22:51:58.821550 (+ 166us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:58.821554 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:58.821642 (+ 88us) server_negotiation.cc:373] Connection header received 0927 22:51:58.821802 (+ 160us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:51:58.821806 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:51:58.821864 (+ 58us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:51:58.821943 (+ 79us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:51:58.822563 (+ 620us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.822713 (+ 150us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.823288 (+ 575us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:58.823799 (+ 511us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:58.823877 (+ 78us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:58.825982 (+ 2105us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:51:58.826003 (+ 21us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:51:58.826007 (+ 4us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:51:58.826033 (+ 26us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:51:58.828287 (+ 2254us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:58.828978 (+ 691us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:58.828982 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:58.828984 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:58.829254 (+ 270us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:58.829668 (+ 414us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:58.829671 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:58.829673 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:58.829832 (+ 159us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:51:58.829999 (+ 167us) server_negotiation.cc:901] Waiting for connection context 0927 22:51:58.830442 (+ 443us) server_negotiation.cc:264] Negotiation successful 0927 22:51:58.830809 (+ 367us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":80} I0927 22:51:58.831845 17689 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } has no permanent_uuid. Determining permanent_uuid... I0927 22:51:58.832988 17690 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.832610 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:58.832803 (+ 193us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.832867 (+ 64us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":72} W0927 22:51:58.833206 17689 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:51:58.849207 17754 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 2 I0927 22:51:58.849807 17756 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.849470 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:58.849605 (+ 135us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.849665 (+ 60us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":49} W0927 22:51:58.849951 17754 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:51:58.896361 17689 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 1 I0927 22:51:58.896968 17690 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.896634 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:58.896797 (+ 163us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.896857 (+ 60us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":55} W0927 22:51:58.897126 17689 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:51:58.951268 17689 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 2 I0927 22:51:58.952178 17690 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.951534 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:58.951828 (+ 294us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.951943 (+ 115us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":103} W0927 22:51:58.952383 17689 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:51:58.961094 17754 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 3 I0927 22:51:58.961974 17756 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:58.961366 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:58.961653 (+ 287us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:58.961766 (+ 113us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":99} W0927 22:51:58.962188 17754 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:51:58.975221 17759 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:51:58.975466 17759 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.60 W0927 22:51:58.975543 17759 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:51:58.977608 17759 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:51:58.977675 17759 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0927 22:51:58.977702 17759 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:51:58.977746 17759 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:51:58.977767 17759 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0927 22:51:58.977807 17759 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.60 I0927 22:51:58.983103 17759 master_main.cc:66] Master server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/wal --ipki_ca_key_size=1024 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.60:11082 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.60.keytab --principal=kudu/127.17.51.60 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/data/info.pb --webserver_interface=127.17.51.60 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:51:58.988219 17759 master_main.cc:73] Initializing master server... I0927 22:51:58.988873 17759 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 50808us Sep 27 22:51:58 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0927 22:51:58.990730 17759 init.cc:339] Logged in from keytab as kudu/127.17.51.60@KRBTEST.COM (short username kudu) I0927 22:51:58.991663 17759 fs_manager.cc:260] Metadata directory not provided I0927 22:51:58.991732 17759 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0927 22:51:58.991938 17759 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/data/instance: No such file or directory (error 2) I0927 22:51:58.992012 17759 server_base.cc:435] Attempting to create new FS layout instead I0927 22:51:58.995110 17759 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/data/instance: uuid: "b6512daf0f244fe7904daf0a2db10237" format_stamp: "Formatted at 2018-09-27 22:51:58 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:58.995645 17759 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/wal/instance: uuid: "b6512daf0f244fe7904daf0a2db10237" format_stamp: "Formatted at 2018-09-27 22:51:58 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:59.001967 17759 fs_manager.cc:495] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s I0927 22:51:59.002993 17759 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0927 22:51:59.003278 17759 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0927 22:51:59.006079 17759 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0927 22:51:59.006150 17759 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/data,/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/wal uuid: "b6512daf0f244fe7904daf0a2db10237" format_stamp: "Formatted at 2018-09-27 22:51:58 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:59.006279 17759 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0927 22:51:59.032522 17689 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 3 I0927 22:51:59.033516 17690 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.032804 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:59.033142 (+ 338us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:59.033262 (+ 120us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":110} W0927 22:51:59.033792 17689 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:51:59.034436 17759 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0927 22:51:59.034773 17759 master_main.cc:76] Starting Master server... I0927 22:51:59.035151 17775 process_memory.cc:182] Process hard memory limit is 11.744253 GB I0927 22:51:59.035225 17775 process_memory.cc:184] Process soft memory limit is 9.395403 GB I0927 22:51:59.035279 17775 process_memory.cc:187] Process memory pressure threshold is 7.046552 GB I0927 22:51:59.048924 17759 rpc_server.cc:201] RPC server started. Bound to: 127.17.51.60:11082 I0927 22:51:59.049386 17759 webserver.cc:173] Starting webserver on 127.17.51.60:0 I0927 22:51:59.049444 17759 webserver.cc:184] Document root disabled I0927 22:51:59.050125 17759 webserver.cc:311] Webserver started. Bound to: http://127.17.51.60:39893/ I0927 22:51:59.050855 17759 server_base.cc:607] Dumped server information to /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-2/data/info.pb I0927 22:51:59.051368 17612 external_mini_cluster.cc:871] Started /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master as pid 17759 I0927 22:51:59.057108 17819 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed WARNING: no policy specified for kudu/127.17.51.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.17.51.1@KRBTEST.COM" created. I0927 22:51:59.064934 17819 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } has no permanent_uuid. Determining permanent_uuid... Sep 27 22:51:59 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.60@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM I0927 22:51:59.081369 17691 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.066842 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40318 0927 22:51:59.067107 (+ 265us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:59.067112 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:59.068096 (+ 984us) server_negotiation.cc:373] Connection header received 0927 22:51:59.070293 (+ 2197us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:51:59.070297 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:51:59.070377 (+ 80us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:51:59.070473 (+ 96us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:51:59.071542 (+ 1069us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.071713 (+ 171us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.072397 (+ 684us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.072901 (+ 504us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.072980 (+ 79us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.075530 (+ 2550us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:51:59.075553 (+ 23us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:51:59.075556 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:51:59.075591 (+ 35us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:51:59.078013 (+ 2422us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.078792 (+ 779us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.078797 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.078799 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.079057 (+ 258us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.079671 (+ 614us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.079675 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.079677 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.079856 (+ 179us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:51:59.080018 (+ 162us) server_negotiation.cc:901] Waiting for connection context 0927 22:51:59.080625 (+ 607us) server_negotiation.cc:264] Negotiation successful 0927 22:51:59.081100 (+ 475us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":80} I0927 22:51:59.081394 17821 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.067220 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:51:59.067970 (+ 750us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:59.068002 (+ 32us) client_negotiation.cc:160] Beginning negotiation 0927 22:51:59.070124 (+ 2122us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:51:59.070645 (+ 521us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:51:59.070653 (+ 8us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:51:59.071102 (+ 449us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:51:59.071268 (+ 166us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.071401 (+ 133us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.071416 (+ 15us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.071829 (+ 413us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.071832 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.072286 (+ 454us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.072296 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.073089 (+ 793us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.073094 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.073215 (+ 121us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.073400 (+ 185us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:51:59.073402 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:51:59.075277 (+ 1875us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:51:59.078260 (+ 2983us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.078265 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.078271 (+ 6us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.078571 (+ 300us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.079316 (+ 745us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.079319 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.079321 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.079401 (+ 80us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.080040 (+ 639us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:51:59.080045 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:51:59.080266 (+ 221us) client_negotiation.cc:695] Sending connection context 0927 22:51:59.080511 (+ 245us) client_negotiation.cc:225] Negotiation successful 0927 22:51:59.080683 (+ 172us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":297,"threads_started":1} I0927 22:51:59.082927 17819 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } has no permanent_uuid. Determining permanent_uuid... Sep 27 22:51:59 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.60@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.17.51.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab. Entry for principal kudu/127.17.51.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab. Entry for principal kudu/127.17.51.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab. Entry for principal kudu/127.17.51.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab. I0927 22:51:59.092222 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.17.51.1:0 --local_ip_for_outbound_sockets=127.17.51.1 --webserver_interface=127.17.51.1 --webserver_port=0 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab --principal=kudu/127.17.51.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --heartbeat_interval_ms=25 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0927 22:51:59.094264 17821 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.083597 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:51:59.083803 (+ 206us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:59.083819 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:51:59.083920 (+ 101us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:51:59.084298 (+ 378us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:51:59.084303 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:51:59.085449 (+ 1146us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:51:59.085528 (+ 79us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.085619 (+ 91us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.085630 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.086056 (+ 426us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.086059 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.086405 (+ 346us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.086414 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.087159 (+ 745us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.087163 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.087241 (+ 78us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.087385 (+ 144us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:51:59.087387 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:51:59.089001 (+ 1614us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:51:59.091841 (+ 2840us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.091845 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.091848 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.092161 (+ 313us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.092847 (+ 686us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.092850 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.092852 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.092921 (+ 69us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.093571 (+ 650us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:51:59.093575 (+ 4us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:51:59.093693 (+ 118us) client_negotiation.cc:695] Sending connection context 0927 22:51:59.093853 (+ 160us) client_negotiation.cc:225] Negotiation successful 0927 22:51:59.093979 (+ 126us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":69} I0927 22:51:59.094652 17758 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.083675 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51114 0927 22:51:59.083922 (+ 247us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:59.083927 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:59.083950 (+ 23us) server_negotiation.cc:373] Connection header received 0927 22:51:59.084017 (+ 67us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:51:59.084022 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:51:59.084137 (+ 115us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:51:59.084239 (+ 102us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:51:59.085771 (+ 1532us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.085950 (+ 179us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.086523 (+ 573us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.087023 (+ 500us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.087117 (+ 94us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.089235 (+ 2118us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:51:59.089274 (+ 39us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:51:59.089278 (+ 4us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:51:59.089308 (+ 30us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:51:59.091632 (+ 2324us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.092391 (+ 759us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.092394 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.092396 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.092671 (+ 275us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.093177 (+ 506us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.093180 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.093182 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.093365 (+ 183us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:51:59.093546 (+ 181us) server_negotiation.cc:901] Waiting for connection context 0927 22:51:59.093974 (+ 428us) server_negotiation.cc:264] Negotiation successful 0927 22:51:59.094383 (+ 409us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":79} I0927 22:51:59.095681 17819 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } has no permanent_uuid. Determining permanent_uuid... Sep 27 22:51:59 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.60@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM I0927 22:51:59.108108 17821 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.096327 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:59.096521 (+ 194us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:59.096535 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:51:59.096628 (+ 93us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:51:59.097269 (+ 641us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:51:59.097273 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:51:59.097621 (+ 348us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:51:59.097689 (+ 68us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.097774 (+ 85us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.097782 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.098254 (+ 472us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.098258 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.098626 (+ 368us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.098635 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.100513 (+ 1878us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.100518 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.100609 (+ 91us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.100754 (+ 145us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:51:59.100756 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:51:59.102471 (+ 1715us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:51:59.105600 (+ 3129us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.105604 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.105607 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.105900 (+ 293us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.106690 (+ 790us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.106693 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.106695 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.106755 (+ 60us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.107385 (+ 630us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:51:59.107388 (+ 3us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:51:59.107507 (+ 119us) client_negotiation.cc:695] Sending connection context 0927 22:51:59.107667 (+ 160us) client_negotiation.cc:225] Negotiation successful 0927 22:51:59.107795 (+ 128us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":2} I0927 22:51:59.108690 17824 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.096307 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42412 0927 22:51:59.096799 (+ 492us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:59.096805 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:59.096831 (+ 26us) server_negotiation.cc:373] Connection header received 0927 22:51:59.096871 (+ 40us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:51:59.096878 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:51:59.097105 (+ 227us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:51:59.097223 (+ 118us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:51:59.097908 (+ 685us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.098145 (+ 237us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.098743 (+ 598us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.100348 (+ 1605us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.100436 (+ 88us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.102723 (+ 2287us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:51:59.102759 (+ 36us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:51:59.102767 (+ 8us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:51:59.102796 (+ 29us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:51:59.105296 (+ 2500us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.106207 (+ 911us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.106213 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.106215 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.106478 (+ 263us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.106936 (+ 458us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.106940 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.106941 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.107170 (+ 229us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:51:59.107345 (+ 175us) server_negotiation.cc:901] Waiting for connection context 0927 22:51:59.107824 (+ 479us) server_negotiation.cc:264] Negotiation successful 0927 22:51:59.108365 (+ 541us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":364,"thread_start_us":230,"threads_started":1} I0927 22:51:59.112975 17819 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237: Bootstrap starting. I0927 22:51:59.114759 17819 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237: No blocks or log segments found. Creating new log. I0927 22:51:59.115511 17819 log.cc:520] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237: Log is configured to *not* fsync() on all Append() calls I0927 22:51:59.117626 17819 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237: No bootstrap required, opened a new log I0927 22:51:59.124179 17819 raft_consensus.cc:304] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:51:59.124410 17819 raft_consensus.cc:330] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:51:59.124505 17819 raft_consensus.cc:605] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6512daf0f244fe7904daf0a2db10237, State: Initialized, Role: FOLLOWER I0927 22:51:59.124855 17819 consensus_queue.cc:226] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:51:59.128738 17819 sys_catalog.cc:335] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.128904 17819 sys_catalog.cc:338] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: This master's current role is: FOLLOWER I0927 22:51:59.128901 17826 sys_catalog.cc:335] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.129307 17826 sys_catalog.cc:338] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: This master's current role is: FOLLOWER I0927 22:51:59.129735 17819 sys_catalog.cc:422] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: configured and running, proceeding with master startup. I0927 22:51:59.132372 17826 raft_consensus.cc:436] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0927 22:51:59.132524 17826 raft_consensus.cc:2700] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237: Snoozing failure detection for 0.105s (starting election) I0927 22:51:59.132879 17826 raft_consensus.cc:458] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:51:59.132972 17759 master_main.cc:79] Master server successfully started. I0927 22:51:59.133653 17826 leader_election.cc:231] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 7c96a1ad26d540babd860e4403b869e4 I0927 22:51:59.133973 17826 leader_election.cc:231] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 2d26965aac2e451c93e86cd8afe389af W0927 22:51:59.137857 17827 catalog_manager.cc:1018] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0927 22:51:59.137956 17827 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0927 22:51:59.138341 17754 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 4 Sep 27 22:51:59 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.61@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM I0927 22:51:59.154124 17756 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.138726 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:59.138974 (+ 248us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:59.138987 (+ 13us) client_negotiation.cc:160] Beginning negotiation 0927 22:51:59.139094 (+ 107us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:51:59.139731 (+ 637us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:51:59.139735 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:51:59.140048 (+ 313us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:51:59.140149 (+ 101us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.140244 (+ 95us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.140253 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.140760 (+ 507us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.140765 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.141218 (+ 453us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.141231 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.142060 (+ 829us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.142076 (+ 16us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.142172 (+ 96us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.142387 (+ 215us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:51:59.142389 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:51:59.145259 (+ 2870us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:51:59.148673 (+ 3414us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.148677 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.148707 (+ 30us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.149229 (+ 522us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.151049 (+ 1820us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.151053 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.151055 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.151185 (+ 130us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.151883 (+ 698us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:51:59.151886 (+ 3us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:51:59.152114 (+ 228us) client_negotiation.cc:695] Sending connection context 0927 22:51:59.152381 (+ 267us) client_negotiation.cc:225] Negotiation successful 0927 22:51:59.152575 (+ 194us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":118} I0927 22:51:59.154882 17824 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.138903 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42414 0927 22:51:59.139237 (+ 334us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:59.139244 (+ 7us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:59.139297 (+ 53us) server_negotiation.cc:373] Connection header received 0927 22:51:59.139390 (+ 93us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:51:59.139394 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:51:59.139542 (+ 148us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:51:59.139715 (+ 173us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:51:59.140386 (+ 671us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.140600 (+ 214us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.141360 (+ 760us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.141895 (+ 535us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.142008 (+ 113us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.145524 (+ 3516us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:51:59.145547 (+ 23us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:51:59.145553 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:51:59.145582 (+ 29us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:51:59.148412 (+ 2830us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.149572 (+ 1160us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.149576 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.149577 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.150814 (+ 1237us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.151473 (+ 659us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.151477 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.151478 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.151652 (+ 174us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:51:59.151845 (+ 193us) server_negotiation.cc:901] Waiting for connection context 0927 22:51:59.152575 (+ 730us) server_negotiation.cc:264] Negotiation successful 0927 22:51:59.154355 (+ 1780us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":98} I0927 22:51:59.158159 17754 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af: Bootstrap starting. I0927 22:51:59.159775 17754 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af: No blocks or log segments found. Creating new log. I0927 22:51:59.161758 17754 log.cc:520] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af: Log is configured to *not* fsync() on all Append() calls I0927 22:51:59.166327 17754 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af: No bootstrap required, opened a new log I0927 22:51:59.173692 17754 raft_consensus.cc:304] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:51:59.173878 17754 raft_consensus.cc:330] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:51:59.173974 17754 raft_consensus.cc:605] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d26965aac2e451c93e86cd8afe389af, State: Initialized, Role: FOLLOWER I0927 22:51:59.174310 17754 consensus_queue.cc:226] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:51:59.176702 17754 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.176898 17754 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [sys.catalog]: This master's current role is: FOLLOWER I0927 22:51:59.177040 17830 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.177537 17830 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [sys.catalog]: This master's current role is: FOLLOWER I0927 22:51:59.177759 17731 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6512daf0f244fe7904daf0a2db10237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d26965aac2e451c93e86cd8afe389af" is_pre_election: true I0927 22:51:59.177980 17731 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af: Snoozing failure detection for 0.101s (vote granted) I0927 22:51:59.178059 17731 raft_consensus.cc:2215] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6512daf0f244fe7904daf0a2db10237 in term 0. I0927 22:51:59.178310 17754 sys_catalog.cc:422] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [sys.catalog]: configured and running, proceeding with master startup. I0927 22:51:59.178900 17768 leader_election.cc:363] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 2d26965aac2e451c93e86cd8afe389af I0927 22:51:59.179002 17768 leader_election.cc:258] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0927 22:51:59.180047 17826 raft_consensus.cc:2700] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237: Snoozing failure detection for 0.107s (election complete) I0927 22:51:59.180158 17826 raft_consensus.cc:2507] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0927 22:51:59.180218 17826 raft_consensus.cc:436] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0927 22:51:59.180269 17826 raft_consensus.cc:2700] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237: Snoozing failure detection for 0.082s (starting election) I0927 22:51:59.180325 17826 raft_consensus.cc:2752] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 0 FOLLOWER]: Advancing to term 1 I0927 22:51:59.180761 17694 master_main.cc:79] Master server successfully started. I0927 22:51:59.182380 17826 raft_consensus.cc:458] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:51:59.183048 17826 leader_election.cc:231] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [CANDIDATE]: Term 1 election: Requesting vote from peer 7c96a1ad26d540babd860e4403b869e4 I0927 22:51:59.183246 17826 leader_election.cc:231] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [CANDIDATE]: Term 1 election: Requesting vote from peer 2d26965aac2e451c93e86cd8afe389af I0927 22:51:59.183938 17731 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6512daf0f244fe7904daf0a2db10237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d26965aac2e451c93e86cd8afe389af" I0927 22:51:59.184056 17731 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [term 0 FOLLOWER]: Advancing to term 1 I0927 22:51:59.184161 17731 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af: Snoozing failure detection for 0.111s (vote granted) I0927 22:51:59.187762 17731 raft_consensus.cc:2215] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6512daf0f244fe7904daf0a2db10237 in term 1. I0927 22:51:59.188625 17768 leader_election.cc:363] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [CANDIDATE]: Term 1 election: Vote granted by peer 2d26965aac2e451c93e86cd8afe389af I0927 22:51:59.188688 17768 leader_election.cc:258] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0927 22:51:59.188906 17826 raft_consensus.cc:2700] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237: Snoozing failure detection for 0.089s (election complete) I0927 22:51:59.188997 17826 raft_consensus.cc:2507] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 1 FOLLOWER]: Leader election won for term 1 W0927 22:51:59.189628 17831 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af: acquiring CA information for follower catalog manager: Not found: root CA entry not found I0927 22:51:59.189863 17826 raft_consensus.cc:572] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 1 LEADER]: Becoming Leader. State: Replica: b6512daf0f244fe7904daf0a2db10237, State: Running, Role: LEADER W0927 22:51:59.189884 17831 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0927 22:51:59.191365 17826 consensus_queue.cc:203] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:51:59.196894 17833 sys_catalog.cc:335] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6512daf0f244fe7904daf0a2db10237. Latest consensus state: current_term: 1 leader_uuid: "b6512daf0f244fe7904daf0a2db10237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.197962 17689 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 4 I0927 22:51:59.199877 17833 sys_catalog.cc:338] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: This master's current role is: LEADER I0927 22:51:59.201565 17835 catalog_manager.cc:972] Loading table and tablet metadata into memory... Sep 27 22:51:59 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.62@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM I0927 22:51:59.214396 17690 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.198368 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:59.198678 (+ 310us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:59.198691 (+ 13us) client_negotiation.cc:160] Beginning negotiation 0927 22:51:59.198779 (+ 88us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:51:59.199656 (+ 877us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:51:59.199659 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:51:59.199977 (+ 318us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:51:59.200048 (+ 71us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.200143 (+ 95us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.200152 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.200752 (+ 600us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.200755 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.201132 (+ 377us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.201141 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.202312 (+ 1171us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.202317 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.202402 (+ 85us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.202535 (+ 133us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:51:59.202538 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:51:59.206839 (+ 4301us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:51:59.211193 (+ 4354us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.211199 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.211202 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.211615 (+ 413us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.212550 (+ 935us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.212554 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.212557 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.212649 (+ 92us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.213580 (+ 931us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:51:59.213584 (+ 4us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:51:59.213731 (+ 147us) client_negotiation.cc:695] Sending connection context 0927 22:51:59.213899 (+ 168us) client_negotiation.cc:225] Negotiation successful 0927 22:51:59.214039 (+ 140us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":192} I0927 22:51:59.215914 17835 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0927 22:51:59.216378 17824 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.198910 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42416 0927 22:51:59.199309 (+ 399us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:59.199312 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:59.199328 (+ 16us) server_negotiation.cc:373] Connection header received 0927 22:51:59.199366 (+ 38us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:51:59.199370 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:51:59.199433 (+ 63us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:51:59.199495 (+ 62us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:51:59.200331 (+ 836us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.200584 (+ 253us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.201304 (+ 720us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.201881 (+ 577us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.201955 (+ 74us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.207290 (+ 5335us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:51:59.207317 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:51:59.207320 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:51:59.207349 (+ 29us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:51:59.210757 (+ 3408us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.211957 (+ 1200us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.211961 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.211963 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.212272 (+ 309us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.212955 (+ 683us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.212959 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.212961 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.213218 (+ 257us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:51:59.215299 (+ 2081us) server_negotiation.cc:901] Waiting for connection context 0927 22:51:59.215509 (+ 210us) server_negotiation.cc:264] Negotiation successful 0927 22:51:59.216052 (+ 543us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":45} I0927 22:51:59.221168 17689 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4: Bootstrap starting. W0927 22:51:59.221935 17710 tablet.cc:1679] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af: Can't schedule compaction. Clean time has not been advanced past its initial value. I0927 22:51:59.224082 17689 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4: No blocks or log segments found. Creating new log. I0927 22:51:59.224903 17731 raft_consensus.cc:1058] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [term 1 FOLLOWER]: Refusing update from remote peer b6512daf0f244fe7904daf0a2db10237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0927 22:51:59.224944 17689 log.cc:520] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4: Log is configured to *not* fsync() on all Append() calls I0927 22:51:59.226022 17833 consensus_queue.cc:962] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0927 22:51:59.230098 17689 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4: No bootstrap required, opened a new log I0927 22:51:59.230258 17830 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6512daf0f244fe7904daf0a2db10237. Latest consensus state: current_term: 1 leader_uuid: "b6512daf0f244fe7904daf0a2db10237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.230430 17830 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [sys.catalog]: This master's current role is: FOLLOWER I0927 22:51:59.236711 17830 sys_catalog.cc:335] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6512daf0f244fe7904daf0a2db10237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.236845 17830 sys_catalog.cc:338] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [sys.catalog]: This master's current role is: FOLLOWER I0927 22:51:59.239056 17689 raft_consensus.cc:304] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:51:59.239321 17689 raft_consensus.cc:330] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:51:59.239394 17689 raft_consensus.cc:605] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c96a1ad26d540babd860e4403b869e4, State: Initialized, Role: FOLLOWER I0927 22:51:59.239771 17689 consensus_queue.cc:226] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } W0927 22:51:59.241147 17769 leader_election.cc:287] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 7c96a1ad26d540babd860e4403b869e4: Timed out: RequestConsensusVote RPC to 127.17.51.62:11080 timed out after 0.105s (SENT) I0927 22:51:59.245512 17689 sys_catalog.cc:335] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.245539 17839 sys_catalog.cc:335] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.245748 17689 sys_catalog.cc:338] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [sys.catalog]: This master's current role is: FOLLOWER I0927 22:51:59.245805 17839 sys_catalog.cc:338] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [sys.catalog]: This master's current role is: FOLLOWER I0927 22:51:59.244987 17833 sys_catalog.cc:335] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6512daf0f244fe7904daf0a2db10237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.245929 17839 raft_consensus.cc:436] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0927 22:51:59.246037 17839 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4: Snoozing failure detection for 0.076s (starting election) I0927 22:51:59.246043 17833 sys_catalog.cc:338] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: This master's current role is: LEADER I0927 22:51:59.246124 17839 raft_consensus.cc:458] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:51:59.249323 17666 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6512daf0f244fe7904daf0a2db10237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c96a1ad26d540babd860e4403b869e4" is_pre_election: true I0927 22:51:59.249615 17665 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b6512daf0f244fe7904daf0a2db10237" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c96a1ad26d540babd860e4403b869e4" I0927 22:51:59.250478 17665 raft_consensus.cc:2190] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate b6512daf0f244fe7904daf0a2db10237 for term 1 because replica is already servicing an update from a current leader or another vote. W0927 22:51:59.250794 17665 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40318 (request call id 3) took 66ms (client timeout 82). W0927 22:51:59.251022 17665 rpcz_store.cc:255] Trace: 0927 22:51:59.183950 (+ 0us) service_pool.cc:163] Inserting onto call queue 0927 22:51:59.184011 (+ 61us) service_pool.cc:222] Handling call 0927 22:51:59.250753 (+ 66742us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":1637120} I0927 22:51:59.251858 17769 leader_election.cc:377] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [CANDIDATE]: Term 1 election: Vote denied by peer 7c96a1ad26d540babd860e4403b869e4. Message: Service unavailable: T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate b6512daf0f244fe7904daf0a2db10237 for term 1 because replica is already servicing an update from a current leader or another vote. I0927 22:51:59.252173 17666 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4: Snoozing failure detection for 0.080s (vote granted) I0927 22:51:59.252245 17666 raft_consensus.cc:2215] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6512daf0f244fe7904daf0a2db10237 in term 0. W0927 22:51:59.253909 17666 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40318 (request call id 2) took 119ms (client timeout 105). W0927 22:51:59.254103 17666 rpcz_store.cc:255] Trace: 0927 22:51:59.134903 (+ 0us) service_pool.cc:163] Inserting onto call queue 0927 22:51:59.134965 (+ 62us) service_pool.cc:222] Handling call 0927 22:51:59.253854 (+118889us) spinlock_profiling.cc:242] Waited 2.62 ms on lock 0x613000011704. stack: 00007fec64cf0ee1 00007fec72184bb9 00007fec6b65945c 00007fec71588a83 00007fec68fe52a2 00007fec6803b90f 00007fec6803a579 00007fec6803e616 00007fec680437a4 00007fec67fa0a7f 00007fec64d16d3e 00007fec6c85c183 00007fec60db5ffc 0927 22:51:59.253887 (+ 33us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":6030080} I0927 22:51:59.254678 17664 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [term 0 FOLLOWER]: Advancing to term 1 I0927 22:51:59.256956 17664 raft_consensus.cc:1058] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [term 1 FOLLOWER]: Refusing update from remote peer b6512daf0f244fe7904daf0a2db10237: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0927 22:51:59.258141 17826 consensus_queue.cc:962] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0927 22:51:59.269170 17839 leader_election.cc:231] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 2d26965aac2e451c93e86cd8afe389af I0927 22:51:59.270686 17731 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c96a1ad26d540babd860e4403b869e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d26965aac2e451c93e86cd8afe389af" is_pre_election: true I0927 22:51:59.270896 17731 raft_consensus.cc:2176] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c96a1ad26d540babd860e4403b869e4 for term 1 because replica is either leader or believes a valid leader to be alive. I0927 22:51:59.271651 17638 leader_election.cc:377] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 2d26965aac2e451c93e86cd8afe389af. Message: Invalid argument: T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7c96a1ad26d540babd860e4403b869e4 for term 1 because replica is either leader or believes a valid leader to be alive. I0927 22:51:59.271911 17839 leader_election.cc:231] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer b6512daf0f244fe7904daf0a2db10237 I0927 22:51:59.273074 17796 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7c96a1ad26d540babd860e4403b869e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6512daf0f244fe7904daf0a2db10237" is_pre_election: true I0927 22:51:59.274464 17796 raft_consensus.cc:2176] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 7c96a1ad26d540babd860e4403b869e4 for term 1 because replica is either leader or believes a valid leader to be alive. I0927 22:51:59.275372 17640 leader_election.cc:377] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [CANDIDATE]: Term 1 pre-election: Vote denied by peer b6512daf0f244fe7904daf0a2db10237. Message: Invalid argument: T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 7c96a1ad26d540babd860e4403b869e4 for term 1 because replica is either leader or believes a valid leader to be alive. I0927 22:51:59.277032 17640 leader_election.cc:258] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0927 22:51:59.279788 17833 sys_catalog.cc:335] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6512daf0f244fe7904daf0a2db10237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.284667 17833 sys_catalog.cc:338] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: This master's current role is: LEADER W0927 22:51:59.285457 17775 tablet.cc:1679] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237: Can't schedule compaction. Clean time has not been advanced past its initial value. I0927 22:51:59.285449 17840 sys_catalog.cc:335] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6512daf0f244fe7904daf0a2db10237. Latest consensus state: current_term: 1 leader_uuid: "b6512daf0f244fe7904daf0a2db10237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.285670 17840 sys_catalog.cc:338] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [sys.catalog]: This master's current role is: FOLLOWER I0927 22:51:59.285847 17689 sys_catalog.cc:422] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [sys.catalog]: configured and running, proceeding with master startup. I0927 22:51:59.289027 17839 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4: Snoozing failure detection for 0.101s (election complete) I0927 22:51:59.289373 17839 raft_consensus.cc:2455] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0927 22:51:59.290144 17629 master_main.cc:79] Master server successfully started. I0927 22:51:59.289278 17842 sys_catalog.cc:335] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b6512daf0f244fe7904daf0a2db10237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:51:59.290777 17842 sys_catalog.cc:338] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [sys.catalog]: This master's current role is: FOLLOWER W0927 22:51:59.296222 17846 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0927 22:51:59.296522 17846 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0927 22:51:59.309314 17835 catalog_manager.cc:870] Generated new certificate authority record I0927 22:51:59.310439 17835 catalog_manager.cc:992] Loading token signing keys... I0927 22:51:59.325546 17835 catalog_manager.cc:3871] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:51:59.358713 17823 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:51:59.359060 17823 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.1 W0927 22:51:59.359177 17823 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:51:59.361398 17823 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:51:59.361531 17823 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:51:59.361582 17823 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:51:59.361660 17823 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.1 W0927 22:51:59.361785 17823 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.51.1 I0927 22:51:59.368363 17823 tablet_server_main.cc:72] Tablet server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.17.51.1:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab --principal=kudu/127.17.51.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data/info.pb --webserver_interface=127.17.51.1 --webserver_port=0 --heartbeat_interval_ms=25 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.17.51.1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:51:59.377492 17823 tablet_server_main.cc:79] Initializing tablet server... I0927 22:51:59.378273 17823 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 51308us Sep 27 22:51:59 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0927 22:51:59.381700 17823 init.cc:339] Logged in from keytab as kudu/127.17.51.1@KRBTEST.COM (short username kudu) I0927 22:51:59.382761 17823 fs_manager.cc:260] Metadata directory not provided I0927 22:51:59.382849 17823 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0927 22:51:59.383095 17823 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0927 22:51:59.383217 17823 server_base.cc:435] Attempting to create new FS layout instead I0927 22:51:59.386291 17823 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data/instance: uuid: "cc9271b36cd048d6bd781e8f0c34b895" format_stamp: "Formatted at 2018-09-27 22:51:59 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:59.386912 17823 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/wal/instance: uuid: "cc9271b36cd048d6bd781e8f0c34b895" format_stamp: "Formatted at 2018-09-27 22:51:59 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:59.392395 17823 fs_manager.cc:495] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s I0927 22:51:59.395087 17823 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0927 22:51:59.395356 17823 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0927 22:51:59.399829 17823 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s I0927 22:51:59.399941 17823 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data,/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/wal uuid: "cc9271b36cd048d6bd781e8f0c34b895" format_stamp: "Formatted at 2018-09-27 22:51:59 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:59.400159 17823 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0927 22:51:59.480947 17823 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0927 22:51:59.481979 17823 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0927 22:51:59.482358 17823 tablet_server_main.cc:84] Starting tablet server... I0927 22:51:59.597915 17823 rpc_server.cc:201] RPC server started. Bound to: 127.17.51.1:39877 I0927 22:51:59.598409 17823 webserver.cc:173] Starting webserver on 127.17.51.1:0 I0927 22:51:59.598464 17823 webserver.cc:184] Document root disabled I0927 22:51:59.599689 17823 webserver.cc:311] Webserver started. Bound to: http://127.17.51.1:33375/ I0927 22:51:59.600400 17823 server_base.cc:607] Dumped server information to /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data/info.pb I0927 22:51:59.602696 17612 external_mini_cluster.cc:871] Started /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver as pid 17823 WARNING: no policy specified for kudu/127.17.51.2@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.17.51.2@KRBTEST.COM" created. I0927 22:51:59.635972 17823 tablet_server_main.cc:87] Tablet server successfully started. I0927 22:51:59.636562 17980 process_memory.cc:182] Process hard memory limit is 11.744253 GB I0927 22:51:59.636615 17980 process_memory.cc:184] Process soft memory limit is 9.395403 GB I0927 22:51:59.636639 17980 process_memory.cc:187] Process memory pressure threshold is 7.046552 GB Sep 27 22:51:59 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.1@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM Sep 27 22:51:59 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.1@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM Sep 27 22:51:59 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.1@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM I0927 22:51:59.654966 17978 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.608680 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.1:41131 0927 22:51:59.632696 (+ 24016us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:59.632699 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:59.634081 (+ 1382us) server_negotiation.cc:373] Connection header received 0927 22:51:59.636628 (+ 2547us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:51:59.636633 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:51:59.636710 (+ 77us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:51:59.636776 (+ 66us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:51:59.638489 (+ 1713us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.638732 (+ 243us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.639504 (+ 772us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.640079 (+ 575us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.640173 (+ 94us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.645984 (+ 5811us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:51:59.646017 (+ 33us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:51:59.646020 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:51:59.646052 (+ 32us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:51:59.650700 (+ 4648us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.651797 (+ 1097us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.651801 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.651803 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.652033 (+ 230us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.652731 (+ 698us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.652735 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.652737 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.652921 (+ 184us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:51:59.653060 (+ 139us) server_negotiation.cc:901] Waiting for connection context 0927 22:51:59.654014 (+ 954us) server_negotiation.cc:264] Negotiation successful 0927 22:51:59.654501 (+ 487us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":23904,"thread_start_us":24285,"threads_started":1} I0927 22:51:59.655561 17975 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.608242 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:51:59.633943 (+ 25701us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:59.633957 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:51:59.636391 (+ 2434us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:51:59.637039 (+ 648us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:51:59.637046 (+ 7us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:51:59.637583 (+ 537us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:51:59.637857 (+ 274us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.638097 (+ 240us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.638112 (+ 15us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.638885 (+ 773us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.638888 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.639382 (+ 494us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.639391 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.640290 (+ 899us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.640295 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.640408 (+ 113us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.640686 (+ 278us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:51:59.640689 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:51:59.645541 (+ 4852us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:51:59.650999 (+ 5458us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.651004 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.651010 (+ 6us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.651430 (+ 420us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.652309 (+ 879us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.652312 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.652314 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.652396 (+ 82us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.653270 (+ 874us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:51:59.653275 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:51:59.653481 (+ 206us) client_negotiation.cc:695] Sending connection context 0927 22:51:59.653878 (+ 397us) client_negotiation.cc:225] Negotiation successful 0927 22:51:59.654352 (+ 474us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":25363,"mutex_wait_us":399,"spinlock_wait_cycles":22016,"thread_start_us":24445,"threads_started":1} I0927 22:51:59.655715 17824 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.635290 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.1:54573 0927 22:51:59.635540 (+ 250us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:59.635545 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:59.636300 (+ 755us) server_negotiation.cc:373] Connection header received 0927 22:51:59.640673 (+ 4373us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:51:59.640678 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:51:59.640759 (+ 81us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:51:59.640835 (+ 76us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:51:59.643388 (+ 2553us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.643566 (+ 178us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.644267 (+ 701us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.644848 (+ 581us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.644915 (+ 67us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.649711 (+ 4796us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:51:59.649744 (+ 33us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:51:59.649747 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:51:59.649781 (+ 34us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:51:59.652183 (+ 2402us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.653009 (+ 826us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.653013 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.653014 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.653290 (+ 276us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.654103 (+ 813us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.654107 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.654108 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.654294 (+ 186us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:51:59.654428 (+ 134us) server_negotiation.cc:901] Waiting for connection context 0927 22:51:59.655048 (+ 620us) server_negotiation.cc:264] Negotiation successful 0927 22:51:59.655475 (+ 427us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":91} I0927 22:51:59.656373 17981 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.634848 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:59.636228 (+ 1380us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:59.636242 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:51:59.640522 (+ 4280us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:51:59.640967 (+ 445us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:51:59.640971 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:51:59.641540 (+ 569us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:51:59.641826 (+ 286us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.643234 (+ 1408us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.643246 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.643699 (+ 453us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.643703 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.644145 (+ 442us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.644155 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.645234 (+ 1079us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.645239 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.645325 (+ 86us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.645505 (+ 180us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:51:59.645507 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:51:59.648653 (+ 3146us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:51:59.652493 (+ 3840us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.652497 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.652499 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.652778 (+ 279us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.653583 (+ 805us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.653586 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.653588 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.653656 (+ 68us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.654571 (+ 915us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:51:59.654574 (+ 3us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:51:59.654702 (+ 128us) client_negotiation.cc:695] Sending connection context 0927 22:51:59.655863 (+ 1161us) client_negotiation.cc:225] Negotiation successful 0927 22:51:59.655995 (+ 132us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1211,"mutex_wait_us":1631,"thread_start_us":1081,"threads_started":1} I0927 22:51:59.657006 17972 heartbeater.cc:346] Connected to a master server at 127.17.51.61:11081 I0927 22:51:59.657150 17972 heartbeater.cc:426] Registering TS with master... I0927 22:51:59.657533 17973 heartbeater.cc:346] Connected to a master server at 127.17.51.60:11082 I0927 22:51:59.657593 17973 heartbeater.cc:426] Registering TS with master... I0927 22:51:59.657845 17972 heartbeater.cc:475] Master 127.17.51.61:11081 requested a full tablet report, sending... I0927 22:51:59.657902 17973 heartbeater.cc:475] Master 127.17.51.60:11082 requested a full tablet report, sending... I0927 22:51:59.659930 17977 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.602442 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:51:59.629471 (+ 27029us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:59.629502 (+ 31us) client_negotiation.cc:160] Beginning negotiation 0927 22:51:59.636391 (+ 6889us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:51:59.637039 (+ 648us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:51:59.637046 (+ 7us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:51:59.637587 (+ 541us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:51:59.637857 (+ 270us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.638100 (+ 243us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.638113 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.640915 (+ 2802us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.640919 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.641792 (+ 873us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:51:59.641807 (+ 15us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.644293 (+ 2486us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.644298 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:51:59.644415 (+ 117us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.644623 (+ 208us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:51:59.644625 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:51:59.646962 (+ 2337us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:51:59.653669 (+ 6707us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.653674 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.653676 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.654059 (+ 383us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.657016 (+ 2957us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:51:59.657020 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:51:59.657023 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:51:59.657117 (+ 94us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:51:59.657858 (+ 741us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:51:59.657861 (+ 3us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:51:59.658045 (+ 184us) client_negotiation.cc:695] Sending connection context 0927 22:51:59.658234 (+ 189us) client_negotiation.cc:225] Negotiation successful 0927 22:51:59.658355 (+ 121us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":26783,"mutex_wait_us":795,"spinlock_wait_cycles":14720,"thread_start_us":30482,"threads_started":1} I0927 22:51:59.659972 17786 ts_manager.cc:81] Registered new tserver with Master: cc9271b36cd048d6bd781e8f0c34b895 (127.17.51.1:39877) I0927 22:51:59.660233 17721 ts_manager.cc:81] Registered new tserver with Master: cc9271b36cd048d6bd781e8f0c34b895 (127.17.51.1:39877) I0927 22:51:59.661365 17786 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.51.1@KRBTEST.COM'} at 127.17.51.1:54573 I0927 22:51:59.661659 17974 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.602034 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.1:37257 0927 22:51:59.609800 (+ 7766us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:59.609804 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:59.629579 (+ 19775us) server_negotiation.cc:373] Connection header received 0927 22:51:59.636630 (+ 7051us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:51:59.636634 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:51:59.636710 (+ 76us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:51:59.636848 (+ 138us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:51:59.638293 (+ 1445us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.640750 (+ 2457us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.641941 (+ 1191us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:51:59.643780 (+ 1839us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:51:59.643865 (+ 85us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:51:59.650289 (+ 6424us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:51:59.650326 (+ 37us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:51:59.650330 (+ 4us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:51:59.650406 (+ 76us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:51:59.653281 (+ 2875us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.656180 (+ 2899us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.656184 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.656186 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.656663 (+ 477us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:51:59.657376 (+ 713us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:51:59.657380 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:51:59.657382 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:51:59.657597 (+ 215us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:51:59.657778 (+ 181us) server_negotiation.cc:901] Waiting for connection context 0927 22:51:59.660611 (+ 2833us) server_negotiation.cc:264] Negotiation successful 0927 22:51:59.661275 (+ 664us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":7630,"thread_start_us":7710,"threads_started":1} I0927 22:51:59.662546 17971 heartbeater.cc:346] Connected to a master server at 127.17.51.62:11080 I0927 22:51:59.662662 17971 heartbeater.cc:426] Registering TS with master... I0927 22:51:59.663105 17971 heartbeater.cc:475] Master 127.17.51.62:11080 requested a full tablet report, sending... Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.17.51.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.2.keytab. Entry for principal kudu/127.17.51.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.2.keytab. Entry for principal kudu/127.17.51.2 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.2.keytab. Entry for principal kudu/127.17.51.2 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.2.keytab. I0927 22:51:59.665032 17656 ts_manager.cc:81] Registered new tserver with Master: cc9271b36cd048d6bd781e8f0c34b895 (127.17.51.1:39877) I0927 22:51:59.665515 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.17.51.2:0 --local_ip_for_outbound_sockets=127.17.51.2 --webserver_interface=127.17.51.2 --webserver_port=0 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.2.keytab --principal=kudu/127.17.51.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --heartbeat_interval_ms=25 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0927 22:51:59.688634 17973 heartbeater.cc:467] Master 127.17.51.60:11082 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:51:59.873185 17982 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:51:59.873423 17982 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.2 W0927 22:51:59.873524 17982 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:51:59.875427 17982 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:51:59.875490 17982 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:51:59.875538 17982 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:51:59.875581 17982 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.2 W0927 22:51:59.875725 17982 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.51.2 I0927 22:51:59.879905 17982 tablet_server_main.cc:72] Tablet server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.17.51.2:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.2.keytab --principal=kudu/127.17.51.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/data/info.pb --webserver_interface=127.17.51.2 --webserver_port=0 --heartbeat_interval_ms=25 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.17.51.2 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:51:59.886157 17982 tablet_server_main.cc:79] Initializing tablet server... I0927 22:51:59.886849 17982 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 51308us Sep 27 22:51:59 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0927 22:51:59.888859 17982 init.cc:339] Logged in from keytab as kudu/127.17.51.2@KRBTEST.COM (short username kudu) I0927 22:51:59.889792 17982 fs_manager.cc:260] Metadata directory not provided I0927 22:51:59.889866 17982 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0927 22:51:59.890048 17982 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/data/instance: No such file or directory (error 2) I0927 22:51:59.890138 17982 server_base.cc:435] Attempting to create new FS layout instead I0927 22:51:59.893195 17982 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/data/instance: uuid: "9aae21a4187e4f1bad11bf9c9d73ba8b" format_stamp: "Formatted at 2018-09-27 22:51:59 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:59.893755 17982 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/wal/instance: uuid: "9aae21a4187e4f1bad11bf9c9d73ba8b" format_stamp: "Formatted at 2018-09-27 22:51:59 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:59.899237 17982 fs_manager.cc:495] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s I0927 22:51:59.901144 17982 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0927 22:51:59.901381 17982 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0927 22:51:59.904230 17982 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0927 22:51:59.904315 17982 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/data,/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/wal uuid: "9aae21a4187e4f1bad11bf9c9d73ba8b" format_stamp: "Formatted at 2018-09-27 22:51:59 on dist-test-slave-dist-test-slave-xppb" I0927 22:51:59.904506 17982 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0927 22:51:59.962049 17982 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0927 22:51:59.962751 17982 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0927 22:51:59.962993 17982 tablet_server_main.cc:84] Starting tablet server... I0927 22:51:59.990857 17982 rpc_server.cc:201] RPC server started. Bound to: 127.17.51.2:34813 I0927 22:51:59.991185 17982 webserver.cc:173] Starting webserver on 127.17.51.2:0 I0927 22:51:59.991235 17982 webserver.cc:184] Document root disabled I0927 22:51:59.991763 17982 webserver.cc:311] Webserver started. Bound to: http://127.17.51.2:42253/ I0927 22:51:59.992269 17982 server_base.cc:607] Dumped server information to /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-1/data/info.pb I0927 22:51:59.996623 17612 external_mini_cluster.cc:871] Started /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver as pid 17982 I0927 22:51:59.997622 18105 process_memory.cc:182] Process hard memory limit is 11.744253 GB I0927 22:52:00.003330 17982 tablet_server_main.cc:87] Tablet server successfully started. I0927 22:52:00.003368 18105 process_memory.cc:184] Process soft memory limit is 9.395403 GB I0927 22:52:00.003408 18105 process_memory.cc:187] Process memory pressure threshold is 7.046552 GB WARNING: no policy specified for kudu/127.17.51.3@KRBTEST.COM; defaulting to no policy Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.2@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.17.51.3@KRBTEST.COM" created. Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.2@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088719, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.2@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM I0927 22:52:00.022773 17974 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.994042 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.2:48229 0927 22:51:59.994232 (+ 190us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:59.994235 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:59.998271 (+ 4036us) server_negotiation.cc:373] Connection header received 0927 22:52:00.003188 (+ 4917us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.003195 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.003306 (+ 111us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.003412 (+ 106us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.004841 (+ 1429us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.005013 (+ 172us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.006762 (+ 1749us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.007284 (+ 522us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.007348 (+ 64us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.011459 (+ 4111us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.011482 (+ 23us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.011485 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.011514 (+ 29us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.014241 (+ 2727us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.018005 (+ 3764us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.018009 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.018011 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.018275 (+ 264us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.018935 (+ 660us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.018939 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.018940 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.019144 (+ 204us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.019288 (+ 144us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.021991 (+ 2703us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.022463 (+ 472us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":66} I0927 22:52:00.024099 17978 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.994472 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.2:33667 0927 22:51:59.994750 (+ 278us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:59.994756 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:59.998371 (+ 3615us) server_negotiation.cc:373] Connection header received 0927 22:52:00.006453 (+ 8082us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.006460 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.006593 (+ 133us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.006671 (+ 78us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.008208 (+ 1537us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.008394 (+ 186us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.009815 (+ 1421us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.010354 (+ 539us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.010418 (+ 64us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.016002 (+ 5584us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.016030 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.016033 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.016082 (+ 49us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.018427 (+ 2345us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.021580 (+ 3153us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.021584 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.021586 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.021846 (+ 260us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.022369 (+ 523us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.022373 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.022374 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.022552 (+ 178us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.022697 (+ 145us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.023426 (+ 729us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.023824 (+ 398us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":98} I0927 22:52:00.024416 18106 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.994672 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:51:59.998175 (+ 3503us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:59.998206 (+ 31us) client_negotiation.cc:160] Beginning negotiation 0927 22:52:00.002926 (+ 4720us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:52:00.003766 (+ 840us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:52:00.003775 (+ 9us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:52:00.004272 (+ 497us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:52:00.004511 (+ 239us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.004683 (+ 172us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:52:00.004699 (+ 16us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.005152 (+ 453us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.005156 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.006625 (+ 1469us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:52:00.006635 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.007932 (+ 1297us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.007940 (+ 8us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.008601 (+ 661us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.008933 (+ 332us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:52:00.008936 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:52:00.011158 (+ 2222us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:52:00.014497 (+ 3339us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:52:00.014502 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:52:00.014508 (+ 6us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:52:00.017764 (+ 3256us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:52:00.018631 (+ 867us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:52:00.018635 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:52:00.018637 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:52:00.018731 (+ 94us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:52:00.019462 (+ 731us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:52:00.019467 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:52:00.021463 (+ 1996us) client_negotiation.cc:695] Sending connection context 0927 22:52:00.023226 (+ 1763us) client_negotiation.cc:225] Negotiation successful 0927 22:52:00.023481 (+ 255us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":3201,"mutex_wait_us":5081,"thread_start_us":2439,"threads_started":1} I0927 22:52:00.025105 17824 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.998663 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.2:50711 0927 22:51:59.998811 (+ 148us) server_negotiation.cc:176] Beginning negotiation 0927 22:51:59.998815 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:51:59.999500 (+ 685us) server_negotiation.cc:373] Connection header received 0927 22:52:00.004639 (+ 5139us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.004644 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.004712 (+ 68us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.004794 (+ 82us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.007464 (+ 2670us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.007683 (+ 219us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.011105 (+ 3422us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.011651 (+ 546us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.011722 (+ 71us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.017957 (+ 6235us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.018002 (+ 45us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.018006 (+ 4us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.018061 (+ 55us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.020690 (+ 2629us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.021505 (+ 815us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.021509 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.021511 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.021776 (+ 265us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.022325 (+ 549us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.022328 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.022330 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.022498 (+ 168us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.022631 (+ 133us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.024383 (+ 1752us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.024861 (+ 478us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":46} I0927 22:52:00.025311 18107 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.996793 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:51:59.998281 (+ 1488us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:59.998296 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:52:00.006273 (+ 7977us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:52:00.006833 (+ 560us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:52:00.006837 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:52:00.007248 (+ 411us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:52:00.007890 (+ 642us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.008042 (+ 152us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:52:00.008052 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.008597 (+ 545us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.008601 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.009290 (+ 689us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:52:00.009299 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.010699 (+ 1400us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.010703 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.010799 (+ 96us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.010990 (+ 191us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:52:00.010993 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:52:00.015714 (+ 4721us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:52:00.019348 (+ 3634us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:52:00.019352 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:52:00.019354 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:52:00.021354 (+ 2000us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:52:00.022087 (+ 733us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:52:00.022090 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:52:00.022093 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:52:00.022147 (+ 54us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:52:00.022883 (+ 736us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:52:00.022886 (+ 3us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:52:00.022993 (+ 107us) client_negotiation.cc:695] Sending connection context 0927 22:52:00.024841 (+ 1848us) client_negotiation.cc:225] Negotiation successful 0927 22:52:00.025002 (+ 161us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1171,"mutex_wait_us":446,"spinlock_wait_cycles":10880,"thread_start_us":1032,"threads_started":1} I0927 22:52:00.026499 18108 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:51:59.998386 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:51:59.999402 (+ 1016us) negotiation.cc:98] Waiting for socket to connect 0927 22:51:59.999413 (+ 11us) client_negotiation.cc:160] Beginning negotiation 0927 22:52:00.004500 (+ 5087us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:52:00.005182 (+ 682us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:52:00.005186 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:52:00.006715 (+ 1529us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:52:00.006831 (+ 116us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.007332 (+ 501us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:52:00.007341 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.007811 (+ 470us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.007814 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.009643 (+ 1829us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:52:00.009651 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.011981 (+ 2330us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.011985 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.012085 (+ 100us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.012237 (+ 152us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:52:00.012239 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:52:00.017546 (+ 5307us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:52:00.020895 (+ 3349us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:52:00.020899 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:52:00.020901 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:52:00.021246 (+ 345us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:52:00.022000 (+ 754us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:52:00.022003 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:52:00.022005 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:52:00.022060 (+ 55us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:52:00.022792 (+ 732us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:52:00.022795 (+ 3us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:52:00.022990 (+ 195us) client_negotiation.cc:695] Sending connection context 0927 22:52:00.025378 (+ 2388us) client_negotiation.cc:225] Negotiation successful 0927 22:52:00.025496 (+ 118us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":3415,"spinlock_wait_cycles":6912,"thread_start_us":784,"threads_started":1} I0927 22:52:00.026777 18102 heartbeater.cc:346] Connected to a master server at 127.17.51.62:11080 I0927 22:52:00.026988 18103 heartbeater.cc:346] Connected to a master server at 127.17.51.61:11081 I0927 22:52:00.026996 18102 heartbeater.cc:426] Registering TS with master... I0927 22:52:00.027076 18103 heartbeater.cc:426] Registering TS with master... I0927 22:52:00.027524 18103 heartbeater.cc:475] Master 127.17.51.61:11081 requested a full tablet report, sending... I0927 22:52:00.027698 18102 heartbeater.cc:475] Master 127.17.51.62:11080 requested a full tablet report, sending... I0927 22:52:00.028532 18104 heartbeater.cc:346] Connected to a master server at 127.17.51.60:11082 I0927 22:52:00.028623 18104 heartbeater.cc:426] Registering TS with master... I0927 22:52:00.028764 17656 ts_manager.cc:81] Registered new tserver with Master: 9aae21a4187e4f1bad11bf9c9d73ba8b (127.17.51.2:34813) I0927 22:52:00.028854 18104 heartbeater.cc:475] Master 127.17.51.60:11082 requested a full tablet report, sending... I0927 22:52:00.029079 17721 ts_manager.cc:81] Registered new tserver with Master: 9aae21a4187e4f1bad11bf9c9d73ba8b (127.17.51.2:34813) I0927 22:52:00.030196 17786 ts_manager.cc:81] Registered new tserver with Master: 9aae21a4187e4f1bad11bf9c9d73ba8b (127.17.51.2:34813) I0927 22:52:00.031150 17786 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.51.2@KRBTEST.COM'} at 127.17.51.2:50711 Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.17.51.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.3.keytab. Entry for principal kudu/127.17.51.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.3.keytab. Entry for principal kudu/127.17.51.3 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.3.keytab. Entry for principal kudu/127.17.51.3 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.3.keytab. I0927 22:52:00.041396 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.17.51.3:0 --local_ip_for_outbound_sockets=127.17.51.3 --webserver_interface=127.17.51.3 --webserver_port=0 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.3.keytab --principal=kudu/127.17.51.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --heartbeat_interval_ms=25 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0927 22:52:00.058034 18104 heartbeater.cc:467] Master 127.17.51.60:11082 was elected leader, sending a full tablet report... I0927 22:52:00.193699 17831 catalog_manager.cc:1016] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af: acquiring CA information for follower catalog manager: success I0927 22:52:00.195230 17831 catalog_manager.cc:1044] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:52:00.215463 18111 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:52:00.215656 18111 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.3 W0927 22:52:00.215721 18111 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:52:00.218317 18111 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:52:00.218389 18111 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:52:00.218425 18111 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:52:00.218463 18111 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.3 W0927 22:52:00.218555 18111 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.51.3 I0927 22:52:00.222347 18111 tablet_server_main.cc:72] Tablet server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.17.51.3:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.3.keytab --principal=kudu/127.17.51.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/data/info.pb --webserver_interface=127.17.51.3 --webserver_port=0 --heartbeat_interval_ms=25 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.17.51.3 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:52:00.226945 18111 tablet_server_main.cc:79] Initializing tablet server... I0927 22:52:00.227516 18111 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 51808us Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088720, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0927 22:52:00.230058 18111 init.cc:339] Logged in from keytab as kudu/127.17.51.3@KRBTEST.COM (short username kudu) I0927 22:52:00.230805 18111 fs_manager.cc:260] Metadata directory not provided I0927 22:52:00.230881 18111 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0927 22:52:00.231040 18111 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/data/instance: No such file or directory (error 2) I0927 22:52:00.231133 18111 server_base.cc:435] Attempting to create new FS layout instead I0927 22:52:00.233827 18111 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/data/instance: uuid: "be00ce656a3f4bdaa010071d3033afb2" format_stamp: "Formatted at 2018-09-27 22:52:00 on dist-test-slave-dist-test-slave-xppb" I0927 22:52:00.234364 18111 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/wal/instance: uuid: "be00ce656a3f4bdaa010071d3033afb2" format_stamp: "Formatted at 2018-09-27 22:52:00 on dist-test-slave-dist-test-slave-xppb" I0927 22:52:00.238823 18111 fs_manager.cc:495] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s I0927 22:52:00.239807 18111 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0927 22:52:00.240006 18111 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0927 22:52:00.242938 18111 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0927 22:52:00.243005 18111 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/data,/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/wal uuid: "be00ce656a3f4bdaa010071d3033afb2" format_stamp: "Formatted at 2018-09-27 22:52:00 on dist-test-slave-dist-test-slave-xppb" I0927 22:52:00.243160 18111 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0927 22:52:00.279184 18111 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0927 22:52:00.279871 18111 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0927 22:52:00.280138 18111 tablet_server_main.cc:84] Starting tablet server... I0927 22:52:00.305732 17846 catalog_manager.cc:1016] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4: acquiring CA information for follower catalog manager: success I0927 22:52:00.307196 17846 catalog_manager.cc:1044] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0927 22:52:00.318078 18111 rpc_server.cc:201] RPC server started. Bound to: 127.17.51.3:38673 I0927 22:52:00.318516 18111 webserver.cc:173] Starting webserver on 127.17.51.3:0 I0927 22:52:00.318572 18111 webserver.cc:184] Document root disabled I0927 22:52:00.319285 18111 webserver.cc:311] Webserver started. Bound to: http://127.17.51.3:46171/ I0927 22:52:00.319983 18111 server_base.cc:607] Dumped server information to /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-2/data/info.pb I0927 22:52:00.322942 18111 tablet_server_main.cc:87] Tablet server successfully started. I0927 22:52:00.324368 18234 process_memory.cc:182] Process hard memory limit is 11.744253 GB I0927 22:52:00.324528 18234 process_memory.cc:184] Process soft memory limit is 9.395403 GB I0927 22:52:00.325083 17612 external_mini_cluster.cc:871] Started /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver as pid 18111 I0927 22:52:00.340850 18234 process_memory.cc:187] Process memory pressure threshold is 7.046552 GB Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM I0927 22:52:00.358682 18237 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.327274 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42418 0927 22:52:00.334429 (+ 7155us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.334432 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.336956 (+ 2524us) server_negotiation.cc:373] Connection header received 0927 22:52:00.341207 (+ 4251us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.341213 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.341316 (+ 103us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.341433 (+ 117us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.343300 (+ 1867us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.343568 (+ 268us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.344431 (+ 863us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.345040 (+ 609us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.345131 (+ 91us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.349666 (+ 4535us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.349698 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.349703 (+ 5us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.349739 (+ 36us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.352702 (+ 2963us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.353626 (+ 924us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.353632 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.353634 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.353959 (+ 325us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.354497 (+ 538us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.354502 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.354504 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.354704 (+ 200us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.354930 (+ 226us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.356439 (+ 1509us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.356948 (+ 509us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":7057,"thread_start_us":6937,"threads_started":1} Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088720, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.3@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088720, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.3@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088720, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.3@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM I0927 22:52:00.369554 17974 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.322327 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.3:47267 0927 22:52:00.322497 (+ 170us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.322500 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.346125 (+ 23625us) server_negotiation.cc:373] Connection header received 0927 22:52:00.350834 (+ 4709us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.350839 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.350918 (+ 79us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.350977 (+ 59us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.352218 (+ 1241us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.352389 (+ 171us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.354312 (+ 1923us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.354835 (+ 523us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.354917 (+ 82us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.361749 (+ 6832us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.361776 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.361780 (+ 4us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.361815 (+ 35us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.364203 (+ 2388us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.365309 (+ 1106us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.365313 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.365315 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.365564 (+ 249us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.367817 (+ 2253us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.367820 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.367822 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.368001 (+ 179us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.368156 (+ 155us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.368902 (+ 746us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.369307 (+ 405us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":56} I0927 22:52:00.370779 18236 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.323879 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:52:00.346001 (+ 22122us) negotiation.cc:98] Waiting for socket to connect 0927 22:52:00.346032 (+ 31us) client_negotiation.cc:160] Beginning negotiation 0927 22:52:00.350598 (+ 4566us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:52:00.351292 (+ 694us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:52:00.351298 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:52:00.351682 (+ 384us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:52:00.351910 (+ 228us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.352052 (+ 142us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:52:00.352078 (+ 26us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.353644 (+ 1566us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.353649 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.354174 (+ 525us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:52:00.354184 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.355264 (+ 1080us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.355270 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.355462 (+ 192us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.355791 (+ 329us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:52:00.355793 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:52:00.361112 (+ 5319us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:52:00.364513 (+ 3401us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:52:00.364519 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:52:00.364524 (+ 5us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:52:00.365057 (+ 533us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:52:00.367218 (+ 2161us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:52:00.367222 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:52:00.367224 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:52:00.367603 (+ 379us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:52:00.368302 (+ 699us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:52:00.368307 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:52:00.368524 (+ 217us) client_negotiation.cc:695] Sending connection context 0927 22:52:00.369849 (+ 1325us) client_negotiation.cc:225] Negotiation successful 0927 22:52:00.370079 (+ 230us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":21844,"mutex_wait_us":668,"thread_start_us":30149,"threads_started":1} I0927 22:52:00.372006 18231 heartbeater.cc:346] Connected to a master server at 127.17.51.62:11080 I0927 22:52:00.372193 18231 heartbeater.cc:426] Registering TS with master... I0927 22:52:00.372324 17978 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.322960 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.3:39311 0927 22:52:00.323151 (+ 191us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.323155 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.348826 (+ 25671us) server_negotiation.cc:373] Connection header received 0927 22:52:00.354572 (+ 5746us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.354576 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.354659 (+ 83us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.354714 (+ 55us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.355823 (+ 1109us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.356033 (+ 210us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.357012 (+ 979us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.357608 (+ 596us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.357674 (+ 66us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.363400 (+ 5726us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.363457 (+ 57us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.363462 (+ 5us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.363509 (+ 47us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.366877 (+ 3368us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.369300 (+ 2423us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.369305 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.369307 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.369601 (+ 294us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.370378 (+ 777us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.370381 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.370383 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.370623 (+ 240us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.370761 (+ 138us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.371478 (+ 717us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.371962 (+ 484us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":68} I0927 22:52:00.372598 17824 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.324966 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.3:48235 0927 22:52:00.325129 (+ 163us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.325133 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.354694 (+ 29561us) server_negotiation.cc:373] Connection header received 0927 22:52:00.354913 (+ 219us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.354917 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.354998 (+ 81us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.355052 (+ 54us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.358508 (+ 3456us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.358824 (+ 316us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.360348 (+ 1524us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.360912 (+ 564us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.360998 (+ 86us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.365087 (+ 4089us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.365116 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.365120 (+ 4us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.365167 (+ 47us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.369105 (+ 3938us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.369945 (+ 840us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.369950 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.369951 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.370239 (+ 288us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.370842 (+ 603us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.370846 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.370847 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.371053 (+ 206us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.371203 (+ 150us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.371835 (+ 632us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.372302 (+ 467us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":45} I0927 22:52:00.372597 18231 heartbeater.cc:475] Master 127.17.51.62:11080 requested a full tablet report, sending... I0927 22:52:00.373371 18239 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.324579 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:52:00.348727 (+ 24148us) negotiation.cc:98] Waiting for socket to connect 0927 22:52:00.348738 (+ 11us) client_negotiation.cc:160] Beginning negotiation 0927 22:52:00.354441 (+ 5703us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:52:00.354864 (+ 423us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:52:00.354867 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:52:00.355297 (+ 430us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:52:00.355412 (+ 115us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.355685 (+ 273us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:52:00.355697 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.356178 (+ 481us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.356181 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.356885 (+ 704us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:52:00.356894 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.357977 (+ 1083us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.357982 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.358081 (+ 99us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.358429 (+ 348us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:52:00.358432 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:52:00.362990 (+ 4558us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:52:00.367156 (+ 4166us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:52:00.367161 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:52:00.367164 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:52:00.367620 (+ 456us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:52:00.369895 (+ 2275us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:52:00.369898 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:52:00.369901 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:52:00.370114 (+ 213us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:52:00.370901 (+ 787us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:52:00.370905 (+ 4us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:52:00.371077 (+ 172us) client_negotiation.cc:695] Sending connection context 0927 22:52:00.372829 (+ 1752us) client_negotiation.cc:225] Negotiation successful 0927 22:52:00.373083 (+ 254us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":24023,"mutex_wait_us":3080,"thread_start_us":21061,"threads_started":1} I0927 22:52:00.373618 17656 ts_manager.cc:81] Registered new tserver with Master: be00ce656a3f4bdaa010071d3033afb2 (127.17.51.3:38673) I0927 22:52:00.375043 18232 heartbeater.cc:346] Connected to a master server at 127.17.51.61:11081 I0927 22:52:00.375142 18232 heartbeater.cc:426] Registering TS with master... I0927 22:52:00.375263 18235 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.325094 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:52:00.354623 (+ 29529us) negotiation.cc:98] Waiting for socket to connect 0927 22:52:00.354639 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:52:00.354802 (+ 163us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:52:00.355217 (+ 415us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:52:00.355220 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:52:00.355928 (+ 708us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:52:00.356218 (+ 290us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.358308 (+ 2090us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:52:00.358317 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.358955 (+ 638us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.358959 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.360193 (+ 1234us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:52:00.360217 (+ 24us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.361083 (+ 866us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.361088 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:52:00.361209 (+ 121us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.361519 (+ 310us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:52:00.361522 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:52:00.364782 (+ 3260us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:52:00.369416 (+ 4634us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:52:00.369419 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:52:00.369422 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:52:00.369698 (+ 276us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:52:00.370569 (+ 871us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:52:00.370573 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:52:00.370575 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:52:00.370632 (+ 57us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:52:00.371363 (+ 731us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:52:00.371366 (+ 3us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:52:00.371491 (+ 125us) client_negotiation.cc:695] Sending connection context 0927 22:52:00.372801 (+ 1310us) client_negotiation.cc:225] Negotiation successful 0927 22:52:00.372963 (+ 162us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":29285,"mutex_wait_us":1455,"thread_start_us":23716,"threads_started":1} I0927 22:52:00.375337 18232 heartbeater.cc:475] Master 127.17.51.61:11081 requested a full tablet report, sending... Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM I0927 22:52:00.377218 17721 ts_manager.cc:81] Registered new tserver with Master: be00ce656a3f4bdaa010071d3033afb2 (127.17.51.3:38673) I0927 22:52:00.377334 18233 heartbeater.cc:346] Connected to a master server at 127.17.51.60:11082 I0927 22:52:00.377399 18233 heartbeater.cc:426] Registering TS with master... I0927 22:52:00.377627 18233 heartbeater.cc:475] Master 127.17.51.60:11082 requested a full tablet report, sending... I0927 22:52:00.378464 17786 ts_manager.cc:81] Registered new tserver with Master: be00ce656a3f4bdaa010071d3033afb2 (127.17.51.3:38673) I0927 22:52:00.379297 17786 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.51.3@KRBTEST.COM'} at 127.17.51.3:48235 I0927 22:52:00.383177 18240 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.361481 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51124 0927 22:52:00.370585 (+ 9104us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.370589 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.370604 (+ 15us) server_negotiation.cc:373] Connection header received 0927 22:52:00.370639 (+ 35us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.370643 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.370697 (+ 54us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.370760 (+ 63us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.371851 (+ 1091us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.372163 (+ 312us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.372841 (+ 678us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.373392 (+ 551us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.373458 (+ 66us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.377222 (+ 3764us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.377247 (+ 25us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.377250 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.377288 (+ 38us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.379894 (+ 2606us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.380914 (+ 1020us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.380919 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.380921 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.381205 (+ 284us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.381689 (+ 484us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.381692 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.381694 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.381878 (+ 184us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.382137 (+ 259us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.382471 (+ 334us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.382897 (+ 426us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":8984,"thread_start_us":10240,"threads_started":1} Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM I0927 22:52:00.396390 17974 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.385037 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40332 0927 22:52:00.385231 (+ 194us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.385235 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.385249 (+ 14us) server_negotiation.cc:373] Connection header received 0927 22:52:00.385383 (+ 134us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.385387 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.385445 (+ 58us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.385525 (+ 80us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.386499 (+ 974us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.386689 (+ 190us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.387335 (+ 646us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.387806 (+ 471us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.387888 (+ 82us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.391312 (+ 3424us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.391333 (+ 21us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.391336 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.391361 (+ 25us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.393658 (+ 2297us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.394397 (+ 739us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.394401 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.394402 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.394639 (+ 237us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.395034 (+ 395us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.395038 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.395039 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.395239 (+ 200us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.395475 (+ 236us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.395806 (+ 331us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.396190 (+ 384us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":72} I0927 22:52:00.405899 18233 heartbeater.cc:467] Master 127.17.51.60:11082 was elected leader, sending a full tablet report... I0927 22:52:00.411167 17824 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.400184 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42424 0927 22:52:00.400328 (+ 144us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.400332 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.400348 (+ 16us) server_negotiation.cc:373] Connection header received 0927 22:52:00.400558 (+ 210us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.400561 (+ 3us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.400624 (+ 63us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.400691 (+ 67us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.401693 (+ 1002us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.401900 (+ 207us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.402544 (+ 644us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.403189 (+ 645us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.403280 (+ 91us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.404617 (+ 1337us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.404660 (+ 43us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.404663 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.404703 (+ 40us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.407557 (+ 2854us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.408345 (+ 788us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.408349 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.408351 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.408653 (+ 302us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.409038 (+ 385us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.409041 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.409043 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.409266 (+ 223us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.410202 (+ 936us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.410505 (+ 303us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.410927 (+ 422us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":46} I0927 22:52:00.412076 17612 external_mini_cluster.cc:499] 3 TS(s) registered with all masters I0927 22:52:00.412266 17612 external_mini_cluster.cc:1042] Attempting to check leaks for /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver pid 17823 Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.1@KRBTEST.COM I0927 22:52:00.430574 18241 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.413328 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:56572 0927 22:52:00.418092 (+ 4764us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.418098 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.418123 (+ 25us) server_negotiation.cc:373] Connection header received 0927 22:52:00.418177 (+ 54us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.418183 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.418356 (+ 173us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.418477 (+ 121us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.419343 (+ 866us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.419533 (+ 190us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.420153 (+ 620us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.420953 (+ 800us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.421039 (+ 86us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.424232 (+ 3193us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.424277 (+ 45us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.424283 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.424326 (+ 43us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.427509 (+ 3183us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.428264 (+ 755us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.428270 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.428272 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.428602 (+ 330us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.429008 (+ 406us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.429012 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.429013 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.429225 (+ 212us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.429500 (+ 275us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.429863 (+ 363us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.430321 (+ 458us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":4583,"thread_start_us":4444,"threads_started":1} I0927 22:52:00.431897 17968 generic_service.cc:169] Checking for leaks (request via RPC) I0927 22:52:00.477109 17612 external_mini_cluster.cc:1001] Killing /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver with pid 17823 W0927 22:52:00.485635 17768 connection.cc:513] server connection from 127.17.51.1:54573 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.485981 17639 connection.cc:513] server connection from 127.17.51.1:37257 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.486016 17703 connection.cc:513] server connection from 127.17.51.1:41131 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.486440 17617 connection.cc:513] client connection to 127.17.51.1:39877 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0927 22:52:00.486654 17612 external_mini_cluster.cc:1042] Attempting to check leaks for /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver pid 17982 Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.2@KRBTEST.COM I0927 22:52:00.499927 18243 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.487352 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:45996 0927 22:52:00.487900 (+ 548us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.487907 (+ 7us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.487933 (+ 26us) server_negotiation.cc:373] Connection header received 0927 22:52:00.487984 (+ 51us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.487990 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.488221 (+ 231us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.488340 (+ 119us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.489364 (+ 1024us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.489598 (+ 234us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.490267 (+ 669us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.491124 (+ 857us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.491215 (+ 91us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.494248 (+ 3033us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.494294 (+ 46us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.494301 (+ 7us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.494333 (+ 32us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.496795 (+ 2462us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.497520 (+ 725us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.497527 (+ 7us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.497529 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.497767 (+ 238us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.498167 (+ 400us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.498170 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.498172 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.498415 (+ 243us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.498669 (+ 254us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.499136 (+ 467us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.499604 (+ 468us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":244,"threads_started":1} I0927 22:52:00.500804 18099 generic_service.cc:169] Checking for leaks (request via RPC) I0927 22:52:00.546174 17612 external_mini_cluster.cc:1001] Killing /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver with pid 17982 W0927 22:52:00.556578 17768 connection.cc:513] server connection from 127.17.51.2:50711 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.556620 17639 connection.cc:513] server connection from 127.17.51.2:48229 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.556619 17704 connection.cc:513] server connection from 127.17.51.2:33667 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.556668 17617 connection.cc:513] client connection to 127.17.51.2:34813 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.556849 17704 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0927 22:52:00.556862 17768 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0927 22:52:00.556869 17639 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0927 22:52:00.557282 17612 external_mini_cluster.cc:1042] Attempting to check leaks for /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver pid 18111 Sep 27 22:52:00 dist-test-slave-dist-test-slave-xppb krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.3@KRBTEST.COM I0927 22:52:00.571674 18245 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.557997 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:56598 0927 22:52:00.558500 (+ 503us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.558507 (+ 7us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.558532 (+ 25us) server_negotiation.cc:373] Connection header received 0927 22:52:00.558590 (+ 58us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.558596 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.558799 (+ 203us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.558911 (+ 112us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.559961 (+ 1050us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.560269 (+ 308us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.560980 (+ 711us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.561912 (+ 932us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.562004 (+ 92us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.565378 (+ 3374us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.565415 (+ 37us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.565422 (+ 7us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.565458 (+ 36us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.568360 (+ 2902us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.569123 (+ 763us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.569129 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.569130 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.569365 (+ 235us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.569837 (+ 472us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.569840 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.569842 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.570046 (+ 204us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.570390 (+ 344us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.570889 (+ 499us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.571385 (+ 496us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":196,"threads_started":1} I0927 22:52:00.572513 18228 generic_service.cc:169] Checking for leaks (request via RPC) I0927 22:52:00.622511 17612 external_mini_cluster.cc:1001] Killing /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver with pid 18111 W0927 22:52:00.631819 17703 connection.cc:513] server connection from 127.17.51.3:39311 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.631814 17771 connection.cc:513] server connection from 127.17.51.3:48235 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.631834 17638 connection.cc:513] server connection from 127.17.51.3:47267 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.632014 17771 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0927 22:52:00.632028 17638 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0927 22:52:00.632032 17703 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0927 22:52:00.632244 17617 connection.cc:513] client connection to 127.17.51.3:38673 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0927 22:52:00.632282 17612 external_mini_cluster.cc:1042] Attempting to check leaks for /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master pid 17629 I0927 22:52:00.642796 17974 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.633117 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40342 0927 22:52:00.633318 (+ 201us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.633323 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.633340 (+ 17us) server_negotiation.cc:373] Connection header received 0927 22:52:00.633496 (+ 156us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.633500 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.633576 (+ 76us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.633669 (+ 93us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.634710 (+ 1041us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.634883 (+ 173us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.635500 (+ 617us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.636016 (+ 516us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.636103 (+ 87us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.637657 (+ 1554us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.637681 (+ 24us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.637685 (+ 4us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.637712 (+ 27us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.639951 (+ 2239us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.640664 (+ 713us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.640668 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.640670 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.640891 (+ 221us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.641294 (+ 403us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.641297 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.641299 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.641477 (+ 178us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.641697 (+ 220us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.642004 (+ 307us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.642520 (+ 516us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":72} I0927 22:52:00.643298 17686 generic_service.cc:169] Checking for leaks (request via RPC) I0927 22:52:00.708026 18248 raft_consensus.cc:436] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b6512daf0f244fe7904daf0a2db10237) I0927 22:52:00.708144 18248 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 7c96a1ad26d540babd860e4403b869e4: Snoozing failure detection for 0.158s (starting election) I0927 22:52:00.708266 17612 external_mini_cluster.cc:1001] Killing /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master with pid 17629 W0927 22:52:00.718854 17617 connection.cc:513] client connection to 127.17.51.62:11080 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.718854 17768 connection.cc:513] server connection from 127.0.0.1:42416 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.718888 17704 connection.cc:513] server connection from 127.0.0.1:51098 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0927 22:52:00.719202 17612 external_mini_cluster.cc:1042] Attempting to check leaks for /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master pid 17694 W0927 22:52:00.719240 17769 connection.cc:513] client connection to 127.17.51.62:11080 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.719374 17769 consensus_peers.cc:435] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 -> Peer 7c96a1ad26d540babd860e4403b869e4 (127.17.51.62:11080): Couldn't send request to peer 7c96a1ad26d540babd860e4403b869e4 for tablet 00000000000000000000000000000000. Status: Network error: failed to read from TLS socket: Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. W0927 22:52:00.719741 17704 connection.cc:513] client connection to 127.17.51.62:11080 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0927 22:52:00.731303 18240 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.720079 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51138 0927 22:52:00.720297 (+ 218us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.720301 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.720344 (+ 43us) server_negotiation.cc:373] Connection header received 0927 22:52:00.720570 (+ 226us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.720574 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.720663 (+ 89us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.720772 (+ 109us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.722230 (+ 1458us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.722424 (+ 194us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.723287 (+ 863us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.723840 (+ 553us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.723913 (+ 73us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.726132 (+ 2219us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.726173 (+ 41us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.726177 (+ 4us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.726205 (+ 28us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.728506 (+ 2301us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.729150 (+ 644us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.729154 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.729156 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.729373 (+ 217us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.729795 (+ 422us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.729798 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.729800 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.729975 (+ 175us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.730211 (+ 236us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.730533 (+ 322us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.731028 (+ 495us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":72} I0927 22:52:00.731797 17751 generic_service.cc:169] Checking for leaks (request via RPC) I0927 22:52:00.734290 17838 sys_catalog.cc:335] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6512daf0f244fe7904daf0a2db10237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:52:00.734637 17838 sys_catalog.cc:338] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: This master's current role is: LEADER I0927 22:52:00.741045 18253 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.735898 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:52:00.740735 (+ 4837us) negotiation.cc:98] Waiting for socket to connect 0927 22:52:00.740849 (+ 114us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":4708,"threads_started":1} W0927 22:52:00.741315 17769 consensus_peers.cc:435] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 -> Peer 7c96a1ad26d540babd860e4403b869e4 (127.17.51.62:11080): Couldn't send request to peer 7c96a1ad26d540babd860e4403b869e4 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I0927 22:52:00.762470 18253 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.761999 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:52:00.762230 (+ 231us) negotiation.cc:98] Waiting for socket to connect 0927 22:52:00.762301 (+ 71us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":94} W0927 22:52:00.762639 17769 consensus_peers.cc:435] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 -> Peer 7c96a1ad26d540babd860e4403b869e4 (127.17.51.62:11080): Couldn't send request to peer 7c96a1ad26d540babd860e4403b869e4 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. I0927 22:52:00.788076 18253 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.787510 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:52:00.787752 (+ 242us) negotiation.cc:98] Waiting for socket to connect 0927 22:52:00.787843 (+ 91us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":62} W0927 22:52:00.788245 17769 consensus_peers.cc:435] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 -> Peer 7c96a1ad26d540babd860e4403b869e4 (127.17.51.62:11080): Couldn't send request to peer 7c96a1ad26d540babd860e4403b869e4 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. I0927 22:52:00.797494 18254 raft_consensus.cc:436] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b6512daf0f244fe7904daf0a2db10237) I0927 22:52:00.797636 18254 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af: Snoozing failure detection for 0.083s (starting election) I0927 22:52:00.797755 18254 raft_consensus.cc:458] T 00000000000000000000000000000000 P 2d26965aac2e451c93e86cd8afe389af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:52:00.797997 17612 external_mini_cluster.cc:1001] Killing /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master with pid 17694 W0927 22:52:00.808828 17617 connection.cc:513] client connection to 127.17.51.61:11081 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.808867 17768 connection.cc:513] client connection to 127.17.51.61:11081 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:52:00.808903 17771 connection.cc:513] server connection from 127.0.0.1:42414 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0927 22:52:00.809181 17612 external_mini_cluster.cc:1042] Attempting to check leaks for /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master pid 17759 I0927 22:52:00.812654 18253 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.812224 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:52:00.812400 (+ 176us) negotiation.cc:98] Waiting for socket to connect 0927 22:52:00.812459 (+ 59us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":72} W0927 22:52:00.812790 17768 consensus_peers.cc:435] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 -> Peer 2d26965aac2e451c93e86cd8afe389af (127.17.51.61:11081): Couldn't send request to peer 2d26965aac2e451c93e86cd8afe389af for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0927 22:52:00.814141 18253 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.813801 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:52:00.813946 (+ 145us) negotiation.cc:98] Waiting for socket to connect 0927 22:52:00.814004 (+ 58us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":51} W0927 22:52:00.814272 17769 consensus_peers.cc:435] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 -> Peer 7c96a1ad26d540babd860e4403b869e4 (127.17.51.62:11080): Couldn't send request to peer 7c96a1ad26d540babd860e4403b869e4 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. I0927 22:52:00.820593 17824 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:52:00.810040 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42442 0927 22:52:00.810288 (+ 248us) server_negotiation.cc:176] Beginning negotiation 0927 22:52:00.810294 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:52:00.810317 (+ 23us) server_negotiation.cc:373] Connection header received 0927 22:52:00.810466 (+ 149us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:52:00.810470 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:52:00.810561 (+ 91us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:52:00.810690 (+ 129us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:52:00.811758 (+ 1068us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.811939 (+ 181us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.812679 (+ 740us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:52:00.813283 (+ 604us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:52:00.813379 (+ 96us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:52:00.815013 (+ 1634us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:52:00.815038 (+ 25us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:52:00.815041 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:52:00.815084 (+ 43us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:52:00.817400 (+ 2316us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.818194 (+ 794us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.818198 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.818200 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.818438 (+ 238us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:52:00.818921 (+ 483us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:52:00.818925 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:52:00.818927 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:52:00.819111 (+ 184us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:52:00.819407 (+ 296us) server_negotiation.cc:901] Waiting for connection context 0927 22:52:00.819845 (+ 438us) server_negotiation.cc:264] Negotiation successful 0927 22:52:00.820351 (+ 506us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":82} I0927 22:52:00.821029 17816 generic_service.cc:169] Checking for leaks (request via RPC) I0927 22:52:00.887662 18251 sys_catalog.cc:335] T 00000000000000000000000000000000 P b6512daf0f244fe7904daf0a2db10237 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b6512daf0f244fe7904daf0a2db10237" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c96a1ad26d540babd860e4403b869e4" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "2d26965aac2e451c93e86cd8afe389af" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "b6512daf0f244fe7904daf0a2db10237" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:52:00.888684 17612 external_mini_cluster.cc:1001] Killing /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master with pid 17759 W0927 22:52:00.899332 17617 connection.cc:513] client connection to 127.17.51.60:11082 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0927 22:54:00.900671 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.62:11080 --webserver_interface=127.17.51.62 --webserver_port=39617 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab --principal=kudu/127.17.51.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:54:01.070560 18257 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:54:01.070776 18257 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.62 W0927 22:54:01.070863 18257 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:54:01.072959 18257 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:54:01.073045 18257 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0927 22:54:01.073112 18257 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:54:01.073158 18257 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:54:01.073205 18257 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0927 22:54:01.073280 18257 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.62 I0927 22:54:01.077508 18257 master_main.cc:66] Master server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.62:11080 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab --principal=kudu/127.17.51.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/data/info.pb --webserver_interface=127.17.51.62 --webserver_port=39617 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:54:01.081976 18257 master_main.cc:73] Initializing master server... I0927 22:54:01.082618 18257 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 112308us F0927 22:54:01.083869 18257 master_main.cc:74] Check failed: _s.ok() Bad status: Runtime error: unable to kinit: unable to login from keytab: Cannot contact any KDC for realm 'KRBTEST.COM' *** Check failure stack trace: *** @ 0x7fd24dd1e31d google::LogMessage::Fail() at ??:0 @ 0x7fd24dd201dd google::LogMessage::SendToLog() at ??:0 @ 0x7fd24dd1de59 google::LogMessage::Flush() at ??:0 @ 0x7fd24dd20c7f google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x521b6f kudu::master::MasterMain() at /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_libc.cc:54 @ 0x521115 main at /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_flag_parser.cc:160 (discriminator 1) @ 0x7fd24b11ef45 __libc_start_main at ??:0 @ 0x42521d (unknown) at ??:0 I0927 22:54:01.213449 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.17.51.1:39877 --local_ip_for_outbound_sockets=127.17.51.1 --webserver_port=33375 --webserver_interface=127.17.51.1 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab --principal=kudu/127.17.51.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --heartbeat_interval_ms=25 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:54:01.375144 18271 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:54:01.375368 18271 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.1 W0927 22:54:01.375471 18271 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:54:01.377490 18271 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:54:01.377583 18271 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:54:01.377635 18271 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:54:01.377699 18271 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.1 W0927 22:54:01.377825 18271 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.51.1 I0927 22:54:01.381832 18271 tablet_server_main.cc:72] Tablet server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.17.51.1:39877 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab --principal=kudu/127.17.51.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/data/info.pb --webserver_interface=127.17.51.1 --webserver_port=33375 --heartbeat_interval_ms=25 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.17.51.1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.NoKdcOnStart.1538088717659548-17612/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:54:01.386798 18271 tablet_server_main.cc:79] Initializing tablet server... I0927 22:54:01.387459 18271 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 112308us F0927 22:54:01.388757 18271 tablet_server_main.cc:80] Check failed: _s.ok() Bad status: Runtime error: unable to kinit: unable to login from keytab: Cannot contact any KDC for realm 'KRBTEST.COM' *** Check failure stack trace: *** @ 0x7fb82c15b31d google::LogMessage::Fail() at ??:0 @ 0x7fb82c15d1dd google::LogMessage::SendToLog() at ??:0 @ 0x7fb82c15ae59 google::LogMessage::Flush() at ??:0 @ 0x7fb82c15dc7f google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x521e98 kudu::tserver::TabletServerMain() at /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_libc.cc:117 @ 0x521465 main at /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_libc.cc:54 @ 0x7fb829d44f45 __libc_start_main at ??:0 @ 0x42556d (unknown) at ??:0 [ OK ] SecurityComponentsFaultsITest.NoKdcOnStart (123850 ms) [ RUN ] SecurityComponentsFaultsITest.KdcRestartsInTheMiddle Loading random data Initializing database '/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Sep 27 22:54:01 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): setting up network... Sep 27 22:54:01 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): listening on fd 9: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Sep 27 22:54:01 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): skipping unrecognized local address family 17 Sep 27 22:54:01 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): skipping unrecognized local address family 17 krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked Sep 27 22:54:01 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): listening on fd 10: udp fe80::42:acff:fe11:2%eth0.0 Sep 27 22:54:01 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): set up 2 sockets Sep 27 22:54:01 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): commencing operation krb5kdc: starting... W0927 22:54:01.895650 17612 mini_kdc.cc:117] Time spent starting KDC: real 0.359s user 0.003s sys 0.011s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal slave/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal slave/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal slave/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Sep 27 22:54:01 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.17.51.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.17.51.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.17.51.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab. Entry for principal kudu/127.17.51.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab. Entry for principal kudu/127.17.51.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab. Entry for principal kudu/127.17.51.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab. I0927 22:54:02.031649 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.62:11080 --webserver_interface=127.17.51.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab --principal=kudu/127.17.51.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:54:02.202883 18297 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:54:02.203109 18297 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.62 W0927 22:54:02.203202 18297 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:54:02.205338 18297 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:54:02.205426 18297 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0927 22:54:02.205487 18297 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:54:02.205538 18297 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:54:02.205580 18297 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0927 22:54:02.205646 18297 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.62 I0927 22:54:02.210047 18297 master_main.cc:66] Master server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.62:11080 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.62.keytab --principal=kudu/127.17.51.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/data/info.pb --webserver_interface=127.17.51.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:54:02.214748 18297 master_main.cc:73] Initializing master server... I0927 22:54:02.215430 18297 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 112808us Sep 27 22:54:02 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0927 22:54:02.217391 18297 init.cc:339] Logged in from keytab as kudu/127.17.51.62@KRBTEST.COM (short username kudu) I0927 22:54:02.218219 18297 fs_manager.cc:260] Metadata directory not provided I0927 22:54:02.218291 18297 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0927 22:54:02.218477 18297 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/data/instance: No such file or directory (error 2) I0927 22:54:02.218547 18297 server_base.cc:435] Attempting to create new FS layout instead I0927 22:54:02.221421 18297 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/data/instance: uuid: "133b07c7153a40f0b5e18b033e7982c0" format_stamp: "Formatted at 2018-09-27 22:54:02 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:02.221952 18297 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/wal/instance: uuid: "133b07c7153a40f0b5e18b033e7982c0" format_stamp: "Formatted at 2018-09-27 22:54:02 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:02.226693 18297 fs_manager.cc:495] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s I0927 22:54:02.227692 18297 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0927 22:54:02.227950 18297 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0927 22:54:02.230398 18297 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0927 22:54:02.230478 18297 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/data,/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/wal uuid: "133b07c7153a40f0b5e18b033e7982c0" format_stamp: "Formatted at 2018-09-27 22:54:02 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:02.230605 18297 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0927 22:54:02.270664 18297 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0927 22:54:02.271050 18297 master_main.cc:76] Starting Master server... I0927 22:54:02.271507 18313 process_memory.cc:182] Process hard memory limit is 11.744253 GB I0927 22:54:02.271579 18313 process_memory.cc:184] Process soft memory limit is 9.395403 GB I0927 22:54:02.271653 18313 process_memory.cc:187] Process memory pressure threshold is 7.046552 GB I0927 22:54:02.282515 18297 rpc_server.cc:201] RPC server started. Bound to: 127.17.51.62:11080 I0927 22:54:02.283021 18297 webserver.cc:173] Starting webserver on 127.17.51.62:0 I0927 22:54:02.283094 18297 webserver.cc:184] Document root disabled I0927 22:54:02.283723 18297 webserver.cc:311] Webserver started. Bound to: http://127.17.51.62:37381/ I0927 22:54:02.284353 18297 server_base.cc:607] Dumped server information to /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-0/data/info.pb I0927 22:54:02.287284 18357 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:54:02.293022 17612 external_mini_cluster.cc:871] Started /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master as pid 18297 I0927 22:54:02.294595 18357 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } has no permanent_uuid. Determining permanent_uuid... Sep 27 22:54:02 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.62@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM WARNING: no policy specified for kudu/127.17.51.61@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.17.51.61@KRBTEST.COM" created. I0927 22:54:02.312674 18358 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.296994 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:02.297949 (+ 955us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.297990 (+ 41us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:02.299370 (+ 1380us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:02.300028 (+ 658us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:02.300033 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:02.300659 (+ 626us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:02.300742 (+ 83us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.300889 (+ 147us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.300904 (+ 15us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.301553 (+ 649us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.301557 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.301994 (+ 437us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.302003 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.303704 (+ 1701us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.303709 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.303801 (+ 92us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.304005 (+ 204us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:02.304008 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:02.305883 (+ 1875us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:02.308816 (+ 2933us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.308825 (+ 9us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.308832 (+ 7us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.309230 (+ 398us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.310231 (+ 1001us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.310234 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.310237 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.310313 (+ 76us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.311240 (+ 927us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:02.311244 (+ 4us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:02.311383 (+ 139us) client_negotiation.cc:695] Sending connection context 0927 22:54:02.311649 (+ 266us) client_negotiation.cc:225] Negotiation successful 0927 22:54:02.311866 (+ 217us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":62,"thread_start_us":413,"threads_started":1} I0927 22:54:02.313343 18359 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.296994 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40376 0927 22:54:02.298239 (+ 1245us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:02.298246 (+ 7us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:02.298288 (+ 42us) server_negotiation.cc:373] Connection header received 0927 22:54:02.299629 (+ 1341us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:02.299637 (+ 8us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:02.299864 (+ 227us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:02.300017 (+ 153us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:02.301046 (+ 1029us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.301410 (+ 364us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.302128 (+ 718us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.303516 (+ 1388us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.303659 (+ 143us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.306175 (+ 2516us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:02.306200 (+ 25us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:02.306206 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:02.306236 (+ 30us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:02.308528 (+ 2292us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.309484 (+ 956us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.309492 (+ 8us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.309494 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.309867 (+ 373us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.310538 (+ 671us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.310543 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.310546 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.310978 (+ 432us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:02.311220 (+ 242us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:02.311923 (+ 703us) server_negotiation.cc:264] Negotiation successful 0927 22:54:02.312687 (+ 764us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":1068,"spinlock_wait_cycles":34560,"thread_start_us":3399,"threads_started":1} I0927 22:54:02.315379 18357 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } has no permanent_uuid. Determining permanent_uuid... I0927 22:54:02.316267 18358 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.315835 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:02.316011 (+ 176us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.316117 (+ 106us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":58} W0927 22:54:02.316485 18357 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.61:11081: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.17.51.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.61.keytab. Entry for principal kudu/127.17.51.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.61.keytab. Entry for principal kudu/127.17.51.61 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.61.keytab. Entry for principal kudu/127.17.51.61 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.61.keytab. I0927 22:54:02.333889 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.61:11081 --webserver_interface=127.17.51.61 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.61.keytab --principal=kudu/127.17.51.61 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0927 22:54:02.335705 18357 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } attempt: 1 I0927 22:54:02.336722 18358 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.336231 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:02.336454 (+ 223us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.336558 (+ 104us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":81} W0927 22:54:02.336872 18357 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.61:11081: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) I0927 22:54:02.385052 18357 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } attempt: 2 I0927 22:54:02.386150 18358 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.385643 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:02.385883 (+ 240us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.385960 (+ 77us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":73} W0927 22:54:02.386328 18357 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.61:11081: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) I0927 22:54:02.497505 18357 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } attempt: 3 I0927 22:54:02.498608 18358 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.498112 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:02.498340 (+ 228us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.498431 (+ 91us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":82} W0927 22:54:02.498823 18357 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.61:11081: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:54:02.503924 18362 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:54:02.504137 18362 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.61 W0927 22:54:02.504251 18362 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:54:02.506353 18362 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:54:02.506438 18362 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0927 22:54:02.506494 18362 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:54:02.506549 18362 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:54:02.506588 18362 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0927 22:54:02.506654 18362 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.61 I0927 22:54:02.510862 18362 master_main.cc:66] Master server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/wal --ipki_ca_key_size=1024 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.61:11081 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.61.keytab --principal=kudu/127.17.51.61 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/data/info.pb --webserver_interface=127.17.51.61 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:54:02.515429 18362 master_main.cc:73] Initializing master server... I0927 22:54:02.516175 18362 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 112808us Sep 27 22:54:02 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0927 22:54:02.518028 18362 init.cc:339] Logged in from keytab as kudu/127.17.51.61@KRBTEST.COM (short username kudu) I0927 22:54:02.518882 18362 fs_manager.cc:260] Metadata directory not provided I0927 22:54:02.518944 18362 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0927 22:54:02.519119 18362 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/data/instance: No such file or directory (error 2) I0927 22:54:02.519194 18362 server_base.cc:435] Attempting to create new FS layout instead I0927 22:54:02.522043 18362 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/data/instance: uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" format_stamp: "Formatted at 2018-09-27 22:54:02 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:02.522549 18362 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/wal/instance: uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" format_stamp: "Formatted at 2018-09-27 22:54:02 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:02.527406 18362 fs_manager.cc:495] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s I0927 22:54:02.528350 18362 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0927 22:54:02.528620 18362 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0927 22:54:02.531214 18362 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0927 22:54:02.531283 18362 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/data,/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/wal uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" format_stamp: "Formatted at 2018-09-27 22:54:02 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:02.531388 18362 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0927 22:54:02.574383 18362 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0927 22:54:02.574713 18362 master_main.cc:76] Starting Master server... I0927 22:54:02.575186 18378 process_memory.cc:182] Process hard memory limit is 11.744253 GB I0927 22:54:02.575279 18378 process_memory.cc:184] Process soft memory limit is 9.395403 GB I0927 22:54:02.575350 18378 process_memory.cc:187] Process memory pressure threshold is 7.046552 GB I0927 22:54:02.586859 18362 rpc_server.cc:201] RPC server started. Bound to: 127.17.51.61:11081 I0927 22:54:02.587254 18362 webserver.cc:173] Starting webserver on 127.17.51.61:0 I0927 22:54:02.587308 18362 webserver.cc:184] Document root disabled I0927 22:54:02.587872 18362 webserver.cc:311] Webserver started. Bound to: http://127.17.51.61:38269/ I0927 22:54:02.588456 18362 server_base.cc:607] Dumped server information to /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-1/data/info.pb I0927 22:54:02.591280 18422 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:54:02.595255 17612 external_mini_cluster.cc:871] Started /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master as pid 18362 I0927 22:54:02.598453 18422 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } has no permanent_uuid. Determining permanent_uuid... WARNING: no policy specified for kudu/127.17.51.60@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.17.51.60@KRBTEST.COM" created. Sep 27 22:54:02 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.61@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM I0927 22:54:02.618782 18424 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.600622 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:02.601355 (+ 733us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.601395 (+ 40us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:02.602829 (+ 1434us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:02.603399 (+ 570us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:02.603405 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:02.603828 (+ 423us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:02.603973 (+ 145us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.604127 (+ 154us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.604140 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.604755 (+ 615us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.604760 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.605247 (+ 487us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.605256 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.606104 (+ 848us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.606112 (+ 8us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.606288 (+ 176us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.606475 (+ 187us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:02.606477 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:02.611111 (+ 4634us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:02.615198 (+ 4087us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.615204 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.615210 (+ 6us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.615550 (+ 340us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.616416 (+ 866us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.616420 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.616422 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.616556 (+ 134us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.617330 (+ 774us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:02.617336 (+ 6us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:02.617571 (+ 235us) client_negotiation.cc:695] Sending connection context 0927 22:54:02.617837 (+ 266us) client_negotiation.cc:225] Negotiation successful 0927 22:54:02.618026 (+ 189us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":311,"threads_started":1} I0927 22:54:02.619030 18359 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.600370 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40386 0927 22:54:02.600643 (+ 273us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:02.600647 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:02.601474 (+ 827us) server_negotiation.cc:373] Connection header received 0927 22:54:02.603014 (+ 1540us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:02.603021 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:02.603129 (+ 108us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:02.603231 (+ 102us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:02.604278 (+ 1047us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.604558 (+ 280us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.605378 (+ 820us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.605913 (+ 535us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.606007 (+ 94us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.611410 (+ 5403us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:02.611460 (+ 50us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:02.611466 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:02.611508 (+ 42us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:02.614946 (+ 3438us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.615783 (+ 837us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.615789 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.615792 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.616189 (+ 397us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.616768 (+ 579us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.616773 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.616776 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.617024 (+ 248us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:02.617286 (+ 262us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:02.617996 (+ 710us) server_negotiation.cc:264] Negotiation successful 0927 22:54:02.618667 (+ 671us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":108} I0927 22:54:02.620689 18422 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } has no permanent_uuid. Determining permanent_uuid... Sep 27 22:54:02 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.61@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM I0927 22:54:02.633558 18424 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.621544 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:02.621953 (+ 409us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.621969 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:02.622076 (+ 107us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:02.622931 (+ 855us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:02.622937 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:02.623332 (+ 395us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:02.623396 (+ 64us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.623531 (+ 135us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.623540 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.624092 (+ 552us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.624096 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.624442 (+ 346us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.624451 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.625631 (+ 1180us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.625635 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.625763 (+ 128us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.625896 (+ 133us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:02.625898 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:02.627765 (+ 1867us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:02.630737 (+ 2972us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.630741 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.630743 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.631022 (+ 279us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.631832 (+ 810us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.631835 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.631837 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.631910 (+ 73us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.632640 (+ 730us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:02.632645 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:02.632778 (+ 133us) client_negotiation.cc:695] Sending connection context 0927 22:54:02.632962 (+ 184us) client_negotiation.cc:225] Negotiation successful 0927 22:54:02.633160 (+ 198us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":106} I0927 22:54:02.634227 18426 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.621534 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51182 0927 22:54:02.622498 (+ 964us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:02.622506 (+ 8us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:02.622535 (+ 29us) server_negotiation.cc:373] Connection header received 0927 22:54:02.622583 (+ 48us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:02.622590 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:02.622782 (+ 192us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:02.622902 (+ 120us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:02.623672 (+ 770us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.623959 (+ 287us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.624593 (+ 634us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.625492 (+ 899us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.625576 (+ 84us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.628106 (+ 2530us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:02.628134 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:02.628140 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:02.628165 (+ 25us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:02.630479 (+ 2314us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.631303 (+ 824us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.631308 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.631310 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.631539 (+ 229us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.632151 (+ 612us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.632155 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.632156 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.632380 (+ 224us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:02.632646 (+ 266us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:02.633227 (+ 581us) server_negotiation.cc:264] Negotiation successful 0927 22:54:02.633984 (+ 757us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":816,"thread_start_us":657,"threads_started":1} I0927 22:54:02.635931 18422 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } has no permanent_uuid. Determining permanent_uuid... Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.17.51.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.60.keytab. Entry for principal kudu/127.17.51.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.60.keytab. Entry for principal kudu/127.17.51.60 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.60.keytab. Entry for principal kudu/127.17.51.60 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.60.keytab. I0927 22:54:02.637043 18424 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.636563 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:02.636744 (+ 181us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.636869 (+ 125us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":64} W0927 22:54:02.637326 18422 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:54:02.637388 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.60:11082 --webserver_interface=127.17.51.60 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.60.keytab --principal=kudu/127.17.51.60 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0927 22:54:02.656534 18422 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 1 I0927 22:54:02.657542 18424 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.657027 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:02.657263 (+ 236us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.657349 (+ 86us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":82} W0927 22:54:02.657764 18422 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:54:02.675029 18357 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } attempt: 4 Sep 27 22:54:02 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.62@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM I0927 22:54:02.687306 18358 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.675632 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:02.675880 (+ 248us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.675894 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:02.675999 (+ 105us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:02.676376 (+ 377us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:02.676379 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:02.676743 (+ 364us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:02.676828 (+ 85us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.676915 (+ 87us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.676923 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.677434 (+ 511us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.677437 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.677794 (+ 357us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.677802 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.678934 (+ 1132us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.678938 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.679021 (+ 83us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.679182 (+ 161us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:02.679184 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:02.681123 (+ 1939us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:02.684113 (+ 2990us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.684120 (+ 7us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.684124 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.684586 (+ 462us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.685356 (+ 770us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.685361 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.685364 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.685550 (+ 186us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.686298 (+ 748us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:02.686303 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:02.686479 (+ 176us) client_negotiation.cc:695] Sending connection context 0927 22:54:02.686752 (+ 273us) client_negotiation.cc:225] Negotiation successful 0927 22:54:02.686927 (+ 175us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":82} I0927 22:54:02.687662 18426 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.675777 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51188 0927 22:54:02.675998 (+ 221us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:02.676005 (+ 7us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:02.676032 (+ 27us) server_negotiation.cc:373] Connection header received 0927 22:54:02.676118 (+ 86us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:02.676124 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:02.676231 (+ 107us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:02.676355 (+ 124us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:02.677040 (+ 685us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.677292 (+ 252us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.677912 (+ 620us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.678791 (+ 879us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.678899 (+ 108us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.681401 (+ 2502us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:02.681426 (+ 25us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:02.681429 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:02.681460 (+ 31us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:02.683825 (+ 2365us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.684824 (+ 999us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.684828 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.684830 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.685111 (+ 281us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.685797 (+ 686us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.685801 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.685803 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.685992 (+ 189us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:02.686196 (+ 204us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:02.686897 (+ 701us) server_negotiation.cc:264] Negotiation successful 0927 22:54:02.687321 (+ 424us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":75} I0927 22:54:02.688808 18357 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } has no permanent_uuid. Determining permanent_uuid... I0927 22:54:02.689798 18358 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.689419 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:02.689608 (+ 189us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.689671 (+ 63us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":68} W0927 22:54:02.690026 18357 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:54:02.705963 18422 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 2 I0927 22:54:02.707098 18424 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.706508 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:02.706767 (+ 259us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.706871 (+ 104us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":79} W0927 22:54:02.707332 18422 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:54:02.753221 18357 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 1 I0927 22:54:02.754355 18358 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.753822 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:02.754107 (+ 285us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.754186 (+ 79us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":80} W0927 22:54:02.754557 18357 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:54:02.808717 18357 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 2 I0927 22:54:02.809829 18358 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.809338 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:02.809572 (+ 234us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.809649 (+ 77us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":90} W0927 22:54:02.810026 18357 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:54:02.814368 18427 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:54:02.814579 18427 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.60 W0927 22:54:02.814683 18427 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:54:02.816900 18427 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:54:02.816989 18427 flags.cc:412] Enabled experimental flag: --ipki_ca_key_size=1024 W0927 22:54:02.817046 18427 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:54:02.817113 18427 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:54:02.817154 18427 flags.cc:412] Enabled experimental flag: --tsk_num_rsa_bits=512 W0927 22:54:02.817230 18427 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.60 I0927 22:54:02.818511 18422 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 3 I0927 22:54:02.819586 18424 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.819022 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:02.819279 (+ 257us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.819369 (+ 90us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":85} W0927 22:54:02.819810 18422 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.17.51.60:11082: Network error: Client connection negotiation failed: client connection to 127.17.51.60:11082: connect: Connection refused (error 111) I0927 22:54:02.822075 18427 master_main.cc:66] Master server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/wal --ipki_ca_key_size=1024 --master_addresses=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.17.51.60:11082 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.60.keytab --principal=kudu/127.17.51.60 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/data/info.pb --webserver_interface=127.17.51.60 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:54:02.826666 18427 master_main.cc:73] Initializing master server... I0927 22:54:02.827344 18427 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 112808us Sep 27 22:54:02 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0927 22:54:02.829146 18427 init.cc:339] Logged in from keytab as kudu/127.17.51.60@KRBTEST.COM (short username kudu) I0927 22:54:02.829924 18427 fs_manager.cc:260] Metadata directory not provided I0927 22:54:02.829993 18427 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0927 22:54:02.830212 18427 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/data/instance: No such file or directory (error 2) I0927 22:54:02.830291 18427 server_base.cc:435] Attempting to create new FS layout instead I0927 22:54:02.833207 18427 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/data/instance: uuid: "9fa6f72caa24460e896f673a01c0fcf2" format_stamp: "Formatted at 2018-09-27 22:54:02 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:02.833735 18427 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/wal/instance: uuid: "9fa6f72caa24460e896f673a01c0fcf2" format_stamp: "Formatted at 2018-09-27 22:54:02 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:02.838690 18427 fs_manager.cc:495] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s I0927 22:54:02.839690 18427 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0927 22:54:02.839942 18427 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0927 22:54:02.842396 18427 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0927 22:54:02.842474 18427 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/data,/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/wal uuid: "9fa6f72caa24460e896f673a01c0fcf2" format_stamp: "Formatted at 2018-09-27 22:54:02 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:02.842612 18427 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0927 22:54:02.864693 18427 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0927 22:54:02.865047 18427 master_main.cc:76] Starting Master server... I0927 22:54:02.865474 18443 process_memory.cc:182] Process hard memory limit is 11.744253 GB I0927 22:54:02.865535 18443 process_memory.cc:184] Process soft memory limit is 9.395403 GB I0927 22:54:02.865587 18443 process_memory.cc:187] Process memory pressure threshold is 7.046552 GB I0927 22:54:02.877265 18427 rpc_server.cc:201] RPC server started. Bound to: 127.17.51.60:11082 I0927 22:54:02.877653 18427 webserver.cc:173] Starting webserver on 127.17.51.60:0 I0927 22:54:02.877718 18427 webserver.cc:184] Document root disabled I0927 22:54:02.878320 18427 webserver.cc:311] Webserver started. Bound to: http://127.17.51.60:37791/ I0927 22:54:02.878940 18427 server_base.cc:607] Dumped server information to /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/master-2/data/info.pb I0927 22:54:02.881798 18487 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:54:02.888208 17612 external_mini_cluster.cc:871] Started /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master as pid 18427 I0927 22:54:02.889045 18487 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } has no permanent_uuid. Determining permanent_uuid... I0927 22:54:02.890211 18357 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 3 Sep 27 22:54:02 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.62@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM Sep 27 22:54:02 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.60@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM WARNING: no policy specified for kudu/127.17.51.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.17.51.1@KRBTEST.COM" created. I0927 22:54:02.908108 18359 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.891001 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40408 0927 22:54:02.891196 (+ 195us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:02.891202 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:02.892212 (+ 1010us) server_negotiation.cc:373] Connection header received 0927 22:54:02.893904 (+ 1692us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:02.893909 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:02.893978 (+ 69us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:02.894069 (+ 91us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:02.895266 (+ 1197us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.895473 (+ 207us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.896393 (+ 920us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.897286 (+ 893us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.897390 (+ 104us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.900569 (+ 3179us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:02.900590 (+ 21us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:02.900595 (+ 5us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:02.900622 (+ 27us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:02.902992 (+ 2370us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.904828 (+ 1836us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.904833 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.904834 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.905185 (+ 351us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.905813 (+ 628us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.905817 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.905818 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.905989 (+ 171us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:02.906242 (+ 253us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:02.907020 (+ 778us) server_negotiation.cc:264] Negotiation successful 0927 22:54:02.907686 (+ 666us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":54} I0927 22:54:02.908174 18488 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.891305 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:02.892095 (+ 790us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.892126 (+ 31us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:02.893746 (+ 1620us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:02.894151 (+ 405us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:02.894160 (+ 9us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:02.894832 (+ 672us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:02.894914 (+ 82us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.895104 (+ 190us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.895123 (+ 19us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.895621 (+ 498us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.895625 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.896262 (+ 637us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.896273 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.897483 (+ 1210us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.897489 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.897709 (+ 220us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.898132 (+ 423us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:02.898136 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:02.900279 (+ 2143us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:02.903279 (+ 3000us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.903288 (+ 9us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.903293 (+ 5us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.904583 (+ 1290us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.905496 (+ 913us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.905499 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.905501 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.905579 (+ 78us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.906269 (+ 690us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:02.906276 (+ 7us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:02.906540 (+ 264us) client_negotiation.cc:695] Sending connection context 0927 22:54:02.906866 (+ 326us) client_negotiation.cc:225] Negotiation successful 0927 22:54:02.907143 (+ 277us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":878,"thread_start_us":324,"threads_started":1} I0927 22:54:02.908522 18358 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.890757 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:02.891203 (+ 446us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.891218 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:02.891324 (+ 106us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:02.894470 (+ 3146us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:02.894473 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:02.894808 (+ 335us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:02.894868 (+ 60us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.894959 (+ 91us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.894968 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.895624 (+ 656us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.895629 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.895998 (+ 369us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.896006 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.897744 (+ 1738us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.897751 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.897872 (+ 121us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.898031 (+ 159us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:02.898035 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:02.899975 (+ 1940us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:02.904449 (+ 4474us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.904454 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.904458 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.905194 (+ 736us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.906119 (+ 925us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.906123 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.906125 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.906185 (+ 60us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.907017 (+ 832us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:02.907021 (+ 4us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:02.907784 (+ 763us) client_negotiation.cc:695] Sending connection context 0927 22:54:02.908006 (+ 222us) client_negotiation.cc:225] Negotiation successful 0927 22:54:02.908183 (+ 177us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":822} I0927 22:54:02.909940 18487 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } has no permanent_uuid. Determining permanent_uuid... I0927 22:54:02.910444 18489 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.891745 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42496 0927 22:54:02.893962 (+ 2217us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:02.893967 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:02.893989 (+ 22us) server_negotiation.cc:373] Connection header received 0927 22:54:02.894123 (+ 134us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:02.894129 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:02.894325 (+ 196us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:02.894489 (+ 164us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:02.895118 (+ 629us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.895485 (+ 367us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.896131 (+ 646us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.897583 (+ 1452us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.897789 (+ 206us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.900381 (+ 2592us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:02.900413 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:02.900422 (+ 9us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:02.900456 (+ 34us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:02.904153 (+ 3697us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.905589 (+ 1436us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.905594 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.905596 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.905850 (+ 254us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.906411 (+ 561us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.906414 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.906416 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.906745 (+ 329us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:02.906956 (+ 211us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:02.909323 (+ 2367us) server_negotiation.cc:264] Negotiation successful 0927 22:54:02.910040 (+ 717us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":245,"server-negotiator.queue_time_us":645,"spinlock_wait_cycles":9472,"thread_start_us":549,"threads_started":1} I0927 22:54:02.915302 18357 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Bootstrap starting. Sep 27 22:54:02 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.60@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM I0927 22:54:02.917327 18357 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: No blocks or log segments found. Creating new log. I0927 22:54:02.918372 18357 log.cc:520] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Log is configured to *not* fsync() on all Append() calls I0927 22:54:02.920068 18357 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: No bootstrap required, opened a new log I0927 22:54:02.924803 18488 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.910563 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:02.911415 (+ 852us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.911430 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:02.911529 (+ 99us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:02.911937 (+ 408us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:02.911942 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:02.912343 (+ 401us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:02.912428 (+ 85us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.912547 (+ 119us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.912558 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.913008 (+ 450us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.913013 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.913513 (+ 500us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.913524 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.914641 (+ 1117us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.914647 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.914778 (+ 131us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.914934 (+ 156us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:02.914938 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:02.917247 (+ 2309us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:02.921664 (+ 4417us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.921668 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.921670 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.921942 (+ 272us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.922959 (+ 1017us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.922964 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.922967 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.923073 (+ 106us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.923742 (+ 669us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:02.923747 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:02.923900 (+ 153us) client_negotiation.cc:695] Sending connection context 0927 22:54:02.924199 (+ 299us) client_negotiation.cc:225] Negotiation successful 0927 22:54:02.924374 (+ 175us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":739} I0927 22:54:02.925113 18426 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.910713 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51204 0927 22:54:02.910927 (+ 214us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:02.910932 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:02.911500 (+ 568us) server_negotiation.cc:373] Connection header received 0927 22:54:02.911661 (+ 161us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:02.911667 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:02.911775 (+ 108us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:02.911889 (+ 114us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:02.912695 (+ 806us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.912888 (+ 193us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.913657 (+ 769us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.914489 (+ 832us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.914588 (+ 99us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.917566 (+ 2978us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:02.917611 (+ 45us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:02.917617 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:02.917658 (+ 41us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:02.921420 (+ 3762us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.922284 (+ 864us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.922291 (+ 7us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.922294 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.922630 (+ 336us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.923288 (+ 658us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.923291 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.923293 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.923494 (+ 201us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:02.923666 (+ 172us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:02.924284 (+ 618us) server_negotiation.cc:264] Negotiation successful 0927 22:54:02.924755 (+ 471us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":69} I0927 22:54:02.926180 18487 sys_catalog.cc:302] member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } has no permanent_uuid. Determining permanent_uuid... I0927 22:54:02.926833 18357 raft_consensus.cc:304] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:54:02.927165 18357 raft_consensus.cc:330] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:54:02.927244 18357 raft_consensus.cc:605] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133b07c7153a40f0b5e18b033e7982c0, State: Initialized, Role: FOLLOWER I0927 22:54:02.928550 18357 consensus_queue.cc:226] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:54:02.928936 18493 raft_consensus.cc:436] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0927 22:54:02.929071 18493 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Snoozing failure detection for 0.090s (starting election) I0927 22:54:02.929258 18493 raft_consensus.cc:458] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:54:02.930274 18493 leader_election.cc:231] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 93ce3bfa2f2a4d00abf078a11b3b7fec I0927 22:54:02.930665 18493 leader_election.cc:231] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 9fa6f72caa24460e896f673a01c0fcf2 I0927 22:54:02.930739 18494 sys_catalog.cc:335] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:54:02.931123 18494 sys_catalog.cc:338] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [sys.catalog]: This master's current role is: FOLLOWER I0927 22:54:02.931819 18357 sys_catalog.cc:335] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } Sep 27 22:54:02 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.60@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM I0927 22:54:02.932034 18357 sys_catalog.cc:338] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [sys.catalog]: This master's current role is: FOLLOWER I0927 22:54:02.932940 18357 sys_catalog.cc:422] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [sys.catalog]: configured and running, proceeding with master startup. I0927 22:54:02.933714 18297 master_main.cc:79] Master server successfully started. W0927 22:54:02.936514 18495 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0927 22:54:02.936605 18495 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.17.51.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab. Entry for principal kudu/127.17.51.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab. Entry for principal kudu/127.17.51.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab. Entry for principal kudu/127.17.51.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab. I0927 22:54:02.938838 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.17.51.1:0 --local_ip_for_outbound_sockets=127.17.51.1 --webserver_interface=127.17.51.1 --webserver_port=0 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab --principal=kudu/127.17.51.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --heartbeat_interval_ms=25 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0927 22:54:02.940203 18488 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.926880 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:02.927083 (+ 203us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.927098 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:02.927188 (+ 90us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:02.927496 (+ 308us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:02.927501 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:02.927832 (+ 331us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:02.927893 (+ 61us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.927968 (+ 75us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.927976 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.928434 (+ 458us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.928439 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.928982 (+ 543us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.928994 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.930138 (+ 1144us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.930143 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.930234 (+ 91us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.930412 (+ 178us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:02.930416 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:02.932782 (+ 2366us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:02.936072 (+ 3290us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.936076 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.936078 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.936413 (+ 335us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.937486 (+ 1073us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:02.937492 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:02.937495 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:02.937579 (+ 84us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:02.938597 (+ 1018us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:02.938602 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:02.938931 (+ 329us) client_negotiation.cc:695] Sending connection context 0927 22:54:02.939437 (+ 506us) client_negotiation.cc:225] Negotiation successful 0927 22:54:02.939782 (+ 345us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":136} I0927 22:54:02.941627 18489 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.926923 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42502 0927 22:54:02.927083 (+ 160us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:02.927099 (+ 16us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:02.927138 (+ 39us) server_negotiation.cc:373] Connection header received 0927 22:54:02.927321 (+ 183us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:02.927327 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:02.927387 (+ 60us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:02.927464 (+ 77us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:02.928111 (+ 647us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.928313 (+ 202us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.929189 (+ 876us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.929973 (+ 784us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.930106 (+ 133us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:02.933131 (+ 3025us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:02.933163 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:02.933169 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:02.933215 (+ 46us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:02.935721 (+ 2506us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.936732 (+ 1011us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.936737 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.936740 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.937153 (+ 413us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:02.937867 (+ 714us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:02.937873 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:02.937876 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:02.938118 (+ 242us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:02.938372 (+ 254us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:02.939578 (+ 1206us) server_negotiation.cc:264] Negotiation successful 0927 22:54:02.941108 (+ 1530us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":216,"server-negotiator.queue_time_us":55} I0927 22:54:02.946123 18487 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Bootstrap starting. I0927 22:54:02.948632 18487 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: No blocks or log segments found. Creating new log. I0927 22:54:02.949687 18487 log.cc:520] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Log is configured to *not* fsync() on all Append() calls I0927 22:54:02.953735 18487 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: No bootstrap required, opened a new log I0927 22:54:02.961097 18487 raft_consensus.cc:304] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:54:02.961398 18487 raft_consensus.cc:330] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:54:02.961474 18487 raft_consensus.cc:605] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa6f72caa24460e896f673a01c0fcf2, State: Initialized, Role: FOLLOWER I0927 22:54:02.961988 18487 consensus_queue.cc:226] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:54:02.968788 18487 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:54:02.969012 18487 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: This master's current role is: FOLLOWER I0927 22:54:02.970147 18464 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133b07c7153a40f0b5e18b033e7982c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa6f72caa24460e896f673a01c0fcf2" is_pre_election: true I0927 22:54:02.970484 18464 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.103s (vote granted) I0927 22:54:02.970609 18464 raft_consensus.cc:2215] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133b07c7153a40f0b5e18b033e7982c0 in term 0. I0927 22:54:02.971251 18487 sys_catalog.cc:422] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: configured and running, proceeding with master startup. I0927 22:54:02.971495 18308 leader_election.cc:363] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 9fa6f72caa24460e896f673a01c0fcf2 I0927 22:54:02.971585 18308 leader_election.cc:258] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0927 22:54:02.972030 18493 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Snoozing failure detection for 0.094s (election complete) I0927 22:54:02.972172 18493 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0927 22:54:02.972285 18493 raft_consensus.cc:436] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0927 22:54:02.972375 18493 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Snoozing failure detection for 0.083s (starting election) I0927 22:54:02.972456 18493 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 0 FOLLOWER]: Advancing to term 1 I0927 22:54:02.968837 18499 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:54:02.973002 18499 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: This master's current role is: FOLLOWER I0927 22:54:02.973548 18427 master_main.cc:79] Master server successfully started. I0927 22:54:02.973156 18500 raft_consensus.cc:436] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0927 22:54:02.973682 18500 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.081s (starting election) I0927 22:54:02.973798 18500 raft_consensus.cc:458] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:54:02.974838 18500 leader_election.cc:231] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 133b07c7153a40f0b5e18b033e7982c0 I0927 22:54:02.975348 18493 raft_consensus.cc:458] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:54:02.976032 18493 leader_election.cc:231] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [CANDIDATE]: Term 1 election: Requesting vote from peer 93ce3bfa2f2a4d00abf078a11b3b7fec I0927 22:54:02.976279 18500 leader_election.cc:231] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 93ce3bfa2f2a4d00abf078a11b3b7fec I0927 22:54:02.977295 18334 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fa6f72caa24460e896f673a01c0fcf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133b07c7153a40f0b5e18b033e7982c0" is_pre_election: true I0927 22:54:02.977644 18334 raft_consensus.cc:2145] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9fa6f72caa24460e896f673a01c0fcf2 in current term 1: Already voted for candidate 133b07c7153a40f0b5e18b033e7982c0 in this term. I0927 22:54:02.978612 18437 leader_election.cc:377] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 133b07c7153a40f0b5e18b033e7982c0. Message: Invalid argument: T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9fa6f72caa24460e896f673a01c0fcf2 in current term 1: Already voted for candidate 133b07c7153a40f0b5e18b033e7982c0 in this term. I0927 22:54:02.978672 18493 leader_election.cc:231] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [CANDIDATE]: Term 1 election: Requesting vote from peer 9fa6f72caa24460e896f673a01c0fcf2 I0927 22:54:02.979707 18464 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133b07c7153a40f0b5e18b033e7982c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fa6f72caa24460e896f673a01c0fcf2" I0927 22:54:02.979858 18464 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 0 FOLLOWER]: Advancing to term 1 I0927 22:54:02.979959 18464 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.090s (vote granted) I0927 22:54:02.983223 18464 raft_consensus.cc:2215] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133b07c7153a40f0b5e18b033e7982c0 in term 1. I0927 22:54:02.983875 18308 leader_election.cc:363] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [CANDIDATE]: Term 1 election: Vote granted by peer 9fa6f72caa24460e896f673a01c0fcf2 I0927 22:54:02.983947 18308 leader_election.cc:258] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0927 22:54:02.984416 18493 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Snoozing failure detection for 0.104s (election complete) I0927 22:54:02.984637 18493 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 1 FOLLOWER]: Leader election won for term 1 I0927 22:54:02.984959 18493 raft_consensus.cc:572] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [term 1 LEADER]: Becoming Leader. State: Replica: 133b07c7153a40f0b5e18b033e7982c0, State: Running, Role: LEADER I0927 22:54:02.985968 18493 consensus_queue.cc:203] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } W0927 22:54:02.989081 18501 catalog_manager.cc:1018] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0927 22:54:02.989245 18501 catalog_manager.cc:560] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0927 22:54:02.992704 18494 sys_catalog.cc:335] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133b07c7153a40f0b5e18b033e7982c0. Latest consensus state: current_term: 1 leader_uuid: "133b07c7153a40f0b5e18b033e7982c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:54:02.993088 18494 sys_catalog.cc:338] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [sys.catalog]: This master's current role is: LEADER I0927 22:54:02.995960 18422 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } attempt: 4 I0927 22:54:02.996888 18504 catalog_manager.cc:972] Loading table and tablet metadata into memory... I0927 22:54:02.999136 18504 catalog_manager.cc:983] Initializing Kudu internal certificate authority... Sep 27 22:54:02 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088842, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.61@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM I0927 22:54:03.016022 18424 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.997032 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:02.997397 (+ 365us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:02.997415 (+ 18us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:02.997539 (+ 124us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:02.998117 (+ 578us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:02.998122 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:02.998598 (+ 476us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:02.998701 (+ 103us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:02.998818 (+ 117us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:02.998829 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.999503 (+ 674us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:02.999508 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.000106 (+ 598us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.000119 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.001659 (+ 1540us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.001664 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.001797 (+ 133us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.001958 (+ 161us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:03.001961 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:03.005050 (+ 3089us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:03.010072 (+ 5022us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.010078 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.010082 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.010511 (+ 429us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.011582 (+ 1071us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.011587 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.011601 (+ 14us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.011681 (+ 80us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.013777 (+ 2096us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:03.013782 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:03.014221 (+ 439us) client_negotiation.cc:695] Sending connection context 0927 22:54:03.014792 (+ 571us) client_negotiation.cc:225] Negotiation successful 0927 22:54:03.015194 (+ 402us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":176} I0927 22:54:03.016324 18464 raft_consensus.cc:1058] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 1 FOLLOWER]: Refusing update from remote peer 133b07c7153a40f0b5e18b033e7982c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0927 22:54:03.017499 18494 consensus_queue.cc:962] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0927 22:54:03.020617 18489 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:02.996674 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42504 0927 22:54:02.996961 (+ 287us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:02.996967 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:02.997649 (+ 682us) server_negotiation.cc:373] Connection header received 0927 22:54:02.997735 (+ 86us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:02.997741 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:02.997835 (+ 94us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:02.997929 (+ 94us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:02.998968 (+ 1039us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:02.999263 (+ 295us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.000307 (+ 1044us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.001221 (+ 914us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.001306 (+ 85us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.005715 (+ 4409us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:03.005751 (+ 36us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:03.005757 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:03.005799 (+ 42us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:03.009696 (+ 3897us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.010886 (+ 1190us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.010891 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.010894 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.011260 (+ 366us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.012185 (+ 925us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.012193 (+ 8us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.012196 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.012904 (+ 708us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:03.017016 (+ 4112us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:03.017769 (+ 753us) server_negotiation.cc:264] Negotiation successful 0927 22:54:03.019610 (+ 1841us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":119} W0927 22:54:03.021492 18306 leader_election.cc:287] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 93ce3bfa2f2a4d00abf078a11b3b7fec: Timed out: RequestConsensusVote RPC to 127.17.51.61:11081 timed out after 0.090s (SENT) I0927 22:54:03.021328 18500 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133b07c7153a40f0b5e18b033e7982c0. Latest consensus state: current_term: 1 leader_uuid: "133b07c7153a40f0b5e18b033e7982c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:54:03.021754 18500 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: This master's current role is: FOLLOWER W0927 22:54:03.022161 18313 tablet.cc:1679] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Can't schedule compaction. Clean time has not been advanced past its initial value. I0927 22:54:03.025984 18422 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Bootstrap starting. I0927 22:54:03.027186 18494 sys_catalog.cc:335] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "133b07c7153a40f0b5e18b033e7982c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:54:03.027432 18494 sys_catalog.cc:338] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [sys.catalog]: This master's current role is: LEADER I0927 22:54:03.028839 18500 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "133b07c7153a40f0b5e18b033e7982c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:54:03.029358 18500 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: This master's current role is: FOLLOWER I0927 22:54:03.032043 18422 tablet_bootstrap.cc:581] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: No blocks or log segments found. Creating new log. I0927 22:54:03.034358 18422 log.cc:520] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Log is configured to *not* fsync() on all Append() calls I0927 22:54:03.037390 18422 tablet_bootstrap.cc:436] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: No bootstrap required, opened a new log I0927 22:54:03.042536 18422 raft_consensus.cc:304] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:54:03.042737 18422 raft_consensus.cc:330] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:54:03.042798 18422 raft_consensus.cc:605] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ce3bfa2f2a4d00abf078a11b3b7fec, State: Initialized, Role: FOLLOWER I0927 22:54:03.043195 18422 consensus_queue.cc:226] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:54:03.046939 18422 sys_catalog.cc:335] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:54:03.047128 18422 sys_catalog.cc:338] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: This master's current role is: FOLLOWER I0927 22:54:03.048104 18422 sys_catalog.cc:422] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: configured and running, proceeding with master startup. I0927 22:54:03.048141 18399 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133b07c7153a40f0b5e18b033e7982c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" is_pre_election: true I0927 22:54:03.048393 18399 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Snoozing failure detection for 0.101s (vote granted) I0927 22:54:03.048512 18399 raft_consensus.cc:2215] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133b07c7153a40f0b5e18b033e7982c0 in term 0. W0927 22:54:03.048631 18399 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51188 (request call id 2) took 117ms (client timeout 90). W0927 22:54:03.048808 18399 rpcz_store.cc:255] Trace: 0927 22:54:02.931415 (+ 0us) service_pool.cc:163] Inserting onto call queue 0927 22:54:02.931491 (+ 76us) service_pool.cc:222] Handling call 0927 22:54:03.048606 (+117115us) inbound_call.cc:157] Queueing success response Metrics: {} I0927 22:54:03.047024 18508 sys_catalog.cc:335] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:54:03.049433 18508 sys_catalog.cc:338] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: This master's current role is: FOLLOWER I0927 22:54:03.054277 18362 master_main.cc:79] Master server successfully started. I0927 22:54:03.054450 18396 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 0 FOLLOWER]: Advancing to term 1 I0927 22:54:03.054155 18398 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "133b07c7153a40f0b5e18b033e7982c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" I0927 22:54:03.054246 18397 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fa6f72caa24460e896f673a01c0fcf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" is_pre_election: true I0927 22:54:03.055905 18504 catalog_manager.cc:870] Generated new certificate authority record I0927 22:54:03.056697 18396 raft_consensus.cc:1058] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 1 FOLLOWER]: Refusing update from remote peer 133b07c7153a40f0b5e18b033e7982c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0927 22:54:03.057538 18397 raft_consensus.cc:2190] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9fa6f72caa24460e896f673a01c0fcf2 for term 1 because replica is already servicing an update from a current leader or another vote. W0927 22:54:03.057566 18436 leader_election.cc:287] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 93ce3bfa2f2a4d00abf078a11b3b7fec: Timed out: RequestConsensusVote RPC to 127.17.51.61:11081 timed out after 0.081s (SENT) I0927 22:54:03.057660 18436 leader_election.cc:258] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0927 22:54:03.057790 18504 catalog_manager.cc:992] Loading token signing keys... W0927 22:54:03.057922 18397 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51204 (request call id 2) took 80ms (client timeout 81). W0927 22:54:03.058101 18397 rpcz_store.cc:255] Trace: 0927 22:54:02.977019 (+ 0us) service_pool.cc:163] Inserting onto call queue 0927 22:54:02.977090 (+ 71us) service_pool.cc:222] Handling call 0927 22:54:03.057844 (+ 80754us) spinlock_profiling.cc:242] Waited 2.55 ms on lock 0x613000011384. stack: 00007f06d554aee1 00007f06e29debb9 00007f06dbeb25da 00007f06e1de2a83 00007f06d983f2a2 00007f06d889590f 00007f06d8894579 00007f06d8898616 00007f06d889d7a4 00007f06d87faa7f 00007f06d5570d3e 00007f06dd0b6183 00007f06d160fffc 0927 22:54:03.057895 (+ 51us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":5860096} I0927 22:54:03.058161 18500 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.098s (election complete) I0927 22:54:03.058240 18500 raft_consensus.cc:2455] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0927 22:54:03.058519 18398 raft_consensus.cc:2190] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 133b07c7153a40f0b5e18b033e7982c0 for term 1 because replica is already servicing an update from a current leader or another vote. W0927 22:54:03.058665 18398 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51188 (request call id 3) took 81ms (client timeout 83). W0927 22:54:03.058815 18398 rpcz_store.cc:255] Trace: 0927 22:54:02.977008 (+ 0us) service_pool.cc:163] Inserting onto call queue 0927 22:54:02.977090 (+ 82us) service_pool.cc:222] Handling call 0927 22:54:03.058592 (+ 81502us) spinlock_profiling.cc:242] Waited 3.66 ms on lock 0x613000011384. stack: 00007f06d554aee1 00007f06e29debb9 00007f06dbeb25da 00007f06e1de2a83 00007f06d983f2a2 00007f06d889590f 00007f06d8894579 00007f06d8898616 00007f06d889d7a4 00007f06d87faa7f 00007f06d5570d3e 00007f06dd0b6183 00007f06d160fffc 0927 22:54:03.058638 (+ 46us) inbound_call.cc:157] Queueing success response Metrics: {"spinlock_wait_cycles":8415488} W0927 22:54:03.059315 18306 leader_election.cc:287] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 93ce3bfa2f2a4d00abf078a11b3b7fec: Timed out: RequestConsensusVote RPC to 127.17.51.61:11081 timed out after 0.083s (SENT) I0927 22:54:03.060024 18494 consensus_queue.cc:962] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0927 22:54:03.073979 18508 sys_catalog.cc:335] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 133b07c7153a40f0b5e18b033e7982c0. Latest consensus state: current_term: 1 leader_uuid: "133b07c7153a40f0b5e18b033e7982c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:54:03.074221 18508 sys_catalog.cc:338] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: This master's current role is: FOLLOWER I0927 22:54:03.075912 18515 sys_catalog.cc:335] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "133b07c7153a40f0b5e18b033e7982c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:54:03.078186 18515 sys_catalog.cc:338] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: This master's current role is: FOLLOWER I0927 22:54:03.082581 18511 catalog_manager.cc:1016] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: acquiring CA information for follower catalog manager: success I0927 22:54:03.082706 18494 sys_catalog.cc:335] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "133b07c7153a40f0b5e18b033e7982c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:54:03.082926 18494 sys_catalog.cc:338] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0 [sys.catalog]: This master's current role is: LEADER W0927 22:54:03.085418 18511 catalog_manager.cc:560] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry I0927 22:54:03.095497 18504 catalog_manager.cc:3871] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:54:03.267218 18497 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:54:03.267603 18497 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.1 W0927 22:54:03.267763 18497 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:54:03.270994 18497 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:54:03.271185 18497 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:54:03.271270 18497 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:54:03.271389 18497 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.1 W0927 22:54:03.271585 18497 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.51.1 I0927 22:54:03.288620 18497 tablet_server_main.cc:72] Tablet server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.17.51.1:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.1.keytab --principal=kudu/127.17.51.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/data/info.pb --webserver_interface=127.17.51.1 --webserver_port=0 --heartbeat_interval_ms=25 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.17.51.1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:54:03.297394 18497 tablet_server_main.cc:79] Initializing tablet server... I0927 22:54:03.298357 18497 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 113308us Sep 27 22:54:03 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088843, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0927 22:54:03.301019 18497 init.cc:339] Logged in from keytab as kudu/127.17.51.1@KRBTEST.COM (short username kudu) I0927 22:54:03.303069 18497 fs_manager.cc:260] Metadata directory not provided I0927 22:54:03.303201 18497 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0927 22:54:03.303455 18497 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/data/instance: No such file or directory (error 2) I0927 22:54:03.303609 18497 server_base.cc:435] Attempting to create new FS layout instead I0927 22:54:03.307682 18497 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/data/instance: uuid: "00f349d4d9e84c4596252016e4bfc620" format_stamp: "Formatted at 2018-09-27 22:54:03 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:03.308514 18497 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/wal/instance: uuid: "00f349d4d9e84c4596252016e4bfc620" format_stamp: "Formatted at 2018-09-27 22:54:03 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:03.318234 18497 fs_manager.cc:495] Time spent creating directory manager: real 0.009s user 0.006s sys 0.002s I0927 22:54:03.319689 18497 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0927 22:54:03.320024 18497 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0927 22:54:03.324028 18497 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0927 22:54:03.324170 18497 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/data,/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/wal uuid: "00f349d4d9e84c4596252016e4bfc620" format_stamp: "Formatted at 2018-09-27 22:54:03 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:03.324424 18497 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0927 22:54:03.417359 18497 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0927 22:54:03.418175 18497 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0927 22:54:03.418467 18497 tablet_server_main.cc:84] Starting tablet server... I0927 22:54:03.503878 18497 rpc_server.cc:201] RPC server started. Bound to: 127.17.51.1:40273 I0927 22:54:03.504302 18497 webserver.cc:173] Starting webserver on 127.17.51.1:0 I0927 22:54:03.504365 18497 webserver.cc:184] Document root disabled I0927 22:54:03.505234 18497 webserver.cc:311] Webserver started. Bound to: http://127.17.51.1:37227/ I0927 22:54:03.505915 18497 server_base.cc:607] Dumped server information to /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/data/info.pb I0927 22:54:03.515184 17612 external_mini_cluster.cc:871] Started /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver as pid 18497 I0927 22:54:03.524770 18644 process_memory.cc:182] Process hard memory limit is 11.744253 GB I0927 22:54:03.524927 18497 tablet_server_main.cc:87] Tablet server successfully started. I0927 22:54:03.524952 18644 process_memory.cc:184] Process soft memory limit is 9.395403 GB I0927 22:54:03.525066 18644 process_memory.cc:187] Process memory pressure threshold is 7.046552 GB WARNING: no policy specified for kudu/127.17.51.2@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.17.51.2@KRBTEST.COM" created. Sep 27 22:54:03 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088843, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.1@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM Sep 27 22:54:03 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088843, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.1@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM Sep 27 22:54:03 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088843, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.1@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM I0927 22:54:03.556075 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:03.507732 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.1:34163 0927 22:54:03.513656 (+ 5924us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:03.513663 (+ 7us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:03.523899 (+ 10236us) server_negotiation.cc:373] Connection header received 0927 22:54:03.538739 (+ 14840us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:03.538746 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:03.538863 (+ 117us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:03.538964 (+ 101us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:03.540484 (+ 1520us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.540759 (+ 275us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.542030 (+ 1271us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.542668 (+ 638us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.542770 (+ 102us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.546184 (+ 3414us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:03.546237 (+ 53us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:03.546243 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:03.546288 (+ 45us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:03.550218 (+ 3930us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.551452 (+ 1234us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.551458 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.551461 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.551875 (+ 414us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.553016 (+ 1141us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.553022 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.553025 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.553329 (+ 304us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:03.553541 (+ 212us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:03.554849 (+ 1308us) server_negotiation.cc:264] Negotiation successful 0927 22:54:03.555661 (+ 812us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5758,"thread_start_us":5628,"threads_started":1} I0927 22:54:03.556934 18642 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:03.508585 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.1:35647 0927 22:54:03.523276 (+ 14691us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:03.523283 (+ 7us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:03.525295 (+ 2012us) server_negotiation.cc:373] Connection header received 0927 22:54:03.539196 (+ 13901us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:03.539203 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:03.539317 (+ 114us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:03.539563 (+ 246us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:03.540754 (+ 1191us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.541018 (+ 264us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.541998 (+ 980us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.542858 (+ 860us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.542954 (+ 96us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.548296 (+ 5342us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:03.548342 (+ 46us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:03.548347 (+ 5us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:03.548387 (+ 40us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:03.551651 (+ 3264us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.552940 (+ 1289us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.552946 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.552950 (+ 4us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.553360 (+ 410us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.554115 (+ 755us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.554121 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.554135 (+ 14us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.554743 (+ 608us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:03.555640 (+ 897us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:03.555873 (+ 233us) server_negotiation.cc:264] Negotiation successful 0927 22:54:03.556556 (+ 683us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":14533,"thread_start_us":14408,"threads_started":1} I0927 22:54:03.558665 18489 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:03.513974 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.1:37893 0927 22:54:03.514321 (+ 347us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:03.514327 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:03.528911 (+ 14584us) server_negotiation.cc:373] Connection header received 0927 22:54:03.538869 (+ 9958us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:03.538875 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:03.538982 (+ 107us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:03.539099 (+ 117us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:03.540553 (+ 1454us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.540750 (+ 197us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.541974 (+ 1224us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.542985 (+ 1011us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.543101 (+ 116us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.550460 (+ 7359us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:03.550506 (+ 46us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:03.550513 (+ 7us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:03.550563 (+ 50us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:03.553920 (+ 3357us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.555082 (+ 1162us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.555087 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.555090 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.555469 (+ 379us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.556200 (+ 731us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.556205 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.556208 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.556499 (+ 291us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:03.556713 (+ 214us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:03.557519 (+ 806us) server_negotiation.cc:264] Negotiation successful 0927 22:54:03.558230 (+ 711us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":187} I0927 22:54:03.558888 18645 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:03.508324 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:03.525211 (+ 16887us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:03.525223 (+ 12us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:03.538529 (+ 13306us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:03.539485 (+ 956us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:03.539493 (+ 8us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:03.540169 (+ 676us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:03.540362 (+ 193us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.540515 (+ 153us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.540527 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.541159 (+ 632us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.541164 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.541813 (+ 649us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.541823 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.543410 (+ 1587us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.543417 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.543555 (+ 138us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.543925 (+ 370us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:03.543928 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:03.547886 (+ 3958us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:03.552004 (+ 4118us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.552010 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.552013 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.552588 (+ 575us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.553675 (+ 1087us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.553680 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.553684 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.553760 (+ 76us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.555144 (+ 1384us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:03.555149 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:03.555303 (+ 154us) client_negotiation.cc:695] Sending connection context 0927 22:54:03.555515 (+ 212us) client_negotiation.cc:225] Negotiation successful 0927 22:54:03.557552 (+ 2037us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":16225,"mutex_wait_us":3688,"thread_start_us":14602,"threads_started":1} I0927 22:54:03.560147 18641 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:03.513966 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:03.528838 (+ 14872us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:03.528853 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:03.538746 (+ 9893us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:03.539260 (+ 514us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:03.539265 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:03.539922 (+ 657us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:03.540098 (+ 176us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.540362 (+ 264us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.540373 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.540908 (+ 535us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.540912 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.541819 (+ 907us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.541827 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.543451 (+ 1624us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.543456 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.543569 (+ 113us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.543937 (+ 368us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:03.543940 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:03.550131 (+ 6191us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:03.554308 (+ 4177us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.554314 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.554318 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.554739 (+ 421us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.555755 (+ 1016us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.555761 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.555764 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.555885 (+ 121us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.556764 (+ 879us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:03.556769 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:03.556978 (+ 209us) client_negotiation.cc:695] Sending connection context 0927 22:54:03.557221 (+ 243us) client_negotiation.cc:225] Negotiation successful 0927 22:54:03.559727 (+ 2506us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":14757,"mutex_wait_us":6382,"spinlock_wait_cycles":13056,"thread_start_us":14504,"threads_started":1} I0927 22:54:03.560662 18638 heartbeater.cc:346] Connected to a master server at 127.17.51.61:11081 I0927 22:54:03.560966 18638 heartbeater.cc:426] Registering TS with master... I0927 22:54:03.561175 18643 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:03.508324 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:03.523754 (+ 15430us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:03.523807 (+ 53us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:03.538529 (+ 14722us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:03.539164 (+ 635us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:03.539175 (+ 11us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:03.539845 (+ 670us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:03.540049 (+ 204us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.540315 (+ 266us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.540330 (+ 15us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.540901 (+ 571us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.540907 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.541812 (+ 905us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.541824 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.542849 (+ 1025us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.542857 (+ 8us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.543027 (+ 170us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.543385 (+ 358us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:03.543389 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:03.545844 (+ 2455us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:03.550561 (+ 4717us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.550569 (+ 8us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.550577 (+ 8us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.551115 (+ 538us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.552251 (+ 1136us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.552259 (+ 8us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.552262 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.552701 (+ 439us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.553752 (+ 1051us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:03.553759 (+ 7us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:03.554103 (+ 344us) client_negotiation.cc:695] Sending connection context 0927 22:54:03.554462 (+ 359us) client_negotiation.cc:225] Negotiation successful 0927 22:54:03.560708 (+ 6246us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":15121,"mutex_wait_us":2548,"thread_start_us":20497,"threads_started":1} I0927 22:54:03.561528 18639 heartbeater.cc:346] Connected to a master server at 127.17.51.60:11082 I0927 22:54:03.561620 18639 heartbeater.cc:426] Registering TS with master... I0927 22:54:03.561936 18639 heartbeater.cc:475] Master 127.17.51.60:11082 requested a full tablet report, sending... I0927 22:54:03.562376 18638 heartbeater.cc:475] Master 127.17.51.61:11081 requested a full tablet report, sending... I0927 22:54:03.562553 18637 heartbeater.cc:346] Connected to a master server at 127.17.51.62:11080 I0927 22:54:03.562649 18637 heartbeater.cc:426] Registering TS with master... I0927 22:54:03.562952 18637 heartbeater.cc:475] Master 127.17.51.62:11080 requested a full tablet report, sending... I0927 22:54:03.563745 18454 ts_manager.cc:81] Registered new tserver with Master: 00f349d4d9e84c4596252016e4bfc620 (127.17.51.1:40273) I0927 22:54:03.563868 18389 ts_manager.cc:81] Registered new tserver with Master: 00f349d4d9e84c4596252016e4bfc620 (127.17.51.1:40273) I0927 22:54:03.564174 18324 ts_manager.cc:81] Registered new tserver with Master: 00f349d4d9e84c4596252016e4bfc620 (127.17.51.1:40273) I0927 22:54:03.565642 18324 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.51.1@KRBTEST.COM'} at 127.17.51.1:34163 Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.17.51.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.2.keytab. Entry for principal kudu/127.17.51.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.2.keytab. Entry for principal kudu/127.17.51.2 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.2.keytab. Entry for principal kudu/127.17.51.2 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.2.keytab. I0927 22:54:03.576722 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.17.51.2:0 --local_ip_for_outbound_sockets=127.17.51.2 --webserver_interface=127.17.51.2 --webserver_port=0 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.2.keytab --principal=kudu/127.17.51.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --heartbeat_interval_ms=25 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0927 22:54:03.592617 18637 heartbeater.cc:467] Master 127.17.51.62:11080 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:54:03.747238 18648 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:54:03.747455 18648 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.2 W0927 22:54:03.747543 18648 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:54:03.749644 18648 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:54:03.749735 18648 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:54:03.749780 18648 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:54:03.749843 18648 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.2 W0927 22:54:03.749966 18648 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.51.2 I0927 22:54:03.754920 18648 tablet_server_main.cc:72] Tablet server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.17.51.2:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.2.keytab --principal=kudu/127.17.51.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/data/info.pb --webserver_interface=127.17.51.2 --webserver_port=0 --heartbeat_interval_ms=25 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.17.51.2 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:54:03.760020 18648 tablet_server_main.cc:79] Initializing tablet server... I0927 22:54:03.760689 18648 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 113308us Sep 27 22:54:03 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088843, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0927 22:54:03.762547 18648 init.cc:339] Logged in from keytab as kudu/127.17.51.2@KRBTEST.COM (short username kudu) I0927 22:54:03.763406 18648 fs_manager.cc:260] Metadata directory not provided I0927 22:54:03.763489 18648 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0927 22:54:03.763655 18648 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/data/instance: No such file or directory (error 2) I0927 22:54:03.763739 18648 server_base.cc:435] Attempting to create new FS layout instead I0927 22:54:03.766464 18648 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/data/instance: uuid: "26b5292fbabd4fbe9b8146fa205ecb22" format_stamp: "Formatted at 2018-09-27 22:54:03 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:03.767122 18648 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/wal/instance: uuid: "26b5292fbabd4fbe9b8146fa205ecb22" format_stamp: "Formatted at 2018-09-27 22:54:03 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:03.772128 18648 fs_manager.cc:495] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s I0927 22:54:03.773196 18648 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0927 22:54:03.773418 18648 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0927 22:54:03.775825 18648 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0927 22:54:03.775883 18648 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/data,/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/wal uuid: "26b5292fbabd4fbe9b8146fa205ecb22" format_stamp: "Formatted at 2018-09-27 22:54:03 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:03.776033 18648 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0927 22:54:03.820425 18648 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0927 22:54:03.821189 18648 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0927 22:54:03.821455 18648 tablet_server_main.cc:84] Starting tablet server... I0927 22:54:03.847777 18648 rpc_server.cc:201] RPC server started. Bound to: 127.17.51.2:41325 I0927 22:54:03.848220 18648 webserver.cc:173] Starting webserver on 127.17.51.2:0 I0927 22:54:03.848280 18648 webserver.cc:184] Document root disabled I0927 22:54:03.848919 18648 webserver.cc:311] Webserver started. Bound to: http://127.17.51.2:37897/ I0927 22:54:03.849512 18648 server_base.cc:607] Dumped server information to /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/data/info.pb I0927 22:54:03.852308 18771 process_memory.cc:182] Process hard memory limit is 11.744253 GB I0927 22:54:03.852414 18648 tablet_server_main.cc:87] Tablet server successfully started. I0927 22:54:03.852571 18771 process_memory.cc:184] Process soft memory limit is 9.395403 GB I0927 22:54:03.852690 18771 process_memory.cc:187] Process memory pressure threshold is 7.046552 GB I0927 22:54:03.858463 17612 external_mini_cluster.cc:871] Started /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver as pid 18648 Sep 27 22:54:03 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088843, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.2@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM Sep 27 22:54:03 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088843, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.2@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM WARNING: no policy specified for kudu/127.17.51.3@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.17.51.3@KRBTEST.COM" created. I0927 22:54:03.881537 18489 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:03.851588 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.2:45075 0927 22:54:03.851802 (+ 214us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:03.851806 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:03.856596 (+ 4790us) server_negotiation.cc:373] Connection header received 0927 22:54:03.860426 (+ 3830us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:03.860433 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:03.860573 (+ 140us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:03.860659 (+ 86us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:03.864379 (+ 3720us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.864685 (+ 306us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.865671 (+ 986us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.866531 (+ 860us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.866630 (+ 99us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.872609 (+ 5979us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:03.872640 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:03.872645 (+ 5us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:03.872686 (+ 41us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:03.876416 (+ 3730us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.877713 (+ 1297us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.877719 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.877722 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.878097 (+ 375us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.878892 (+ 795us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.878897 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.878900 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.879181 (+ 281us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:03.879367 (+ 186us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:03.880556 (+ 1189us) server_negotiation.cc:264] Negotiation successful 0927 22:54:03.881217 (+ 661us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":83} Sep 27 22:54:03 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088843, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.2@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM I0927 22:54:03.883599 18772 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:03.852862 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:03.856514 (+ 3652us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:03.856527 (+ 13us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:03.860284 (+ 3757us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:03.860834 (+ 550us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:03.860839 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:03.864006 (+ 3167us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:03.864134 (+ 128us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.864230 (+ 96us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.864241 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.864920 (+ 679us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.864929 (+ 9us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.865519 (+ 590us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.865530 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.869050 (+ 3520us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.869072 (+ 22us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.869196 (+ 124us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.869449 (+ 253us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:03.869453 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:03.872138 (+ 2685us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:03.876725 (+ 4587us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.876732 (+ 7us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.876746 (+ 14us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.877321 (+ 575us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.878422 (+ 1101us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.878427 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.878430 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.878541 (+ 111us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.879555 (+ 1014us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:03.879562 (+ 7us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:03.879893 (+ 331us) client_negotiation.cc:695] Sending connection context 0927 22:54:03.880369 (+ 476us) client_negotiation.cc:225] Negotiation successful 0927 22:54:03.882473 (+ 2104us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":2335,"thread_start_us":512,"threads_started":1} I0927 22:54:03.883705 18773 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:03.852863 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:03.854276 (+ 1413us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:03.854313 (+ 37us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:03.860020 (+ 5707us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:03.860773 (+ 753us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:03.860782 (+ 9us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:03.863355 (+ 2573us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:03.863589 (+ 234us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.863776 (+ 187us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.863799 (+ 23us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.864434 (+ 635us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.864439 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.865251 (+ 812us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.865263 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.866726 (+ 1463us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.866733 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.866904 (+ 171us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.867189 (+ 285us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:03.867192 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:03.869761 (+ 2569us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:03.876856 (+ 7095us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.876861 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.876865 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.877795 (+ 930us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.878904 (+ 1109us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.878909 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.878913 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.878993 (+ 80us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.880099 (+ 1106us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:03.880104 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:03.882304 (+ 2200us) client_negotiation.cc:695] Sending connection context 0927 22:54:03.883132 (+ 828us) client_negotiation.cc:225] Negotiation successful 0927 22:54:03.883289 (+ 157us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":5399,"spinlock_wait_cycles":20864,"thread_start_us":604,"threads_started":1} I0927 22:54:03.885699 18770 heartbeater.cc:346] Connected to a master server at 127.17.51.60:11082 I0927 22:54:03.885870 18770 heartbeater.cc:426] Registering TS with master... I0927 22:54:03.886328 18770 heartbeater.cc:475] Master 127.17.51.60:11082 requested a full tablet report, sending... I0927 22:54:03.887466 18642 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:03.851415 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.2:58565 0927 22:54:03.851637 (+ 222us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:03.851643 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:03.854403 (+ 2760us) server_negotiation.cc:373] Connection header received 0927 22:54:03.860232 (+ 5829us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:03.860239 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:03.860346 (+ 107us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:03.860447 (+ 101us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:03.863940 (+ 3493us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.864278 (+ 338us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.865414 (+ 1136us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.866274 (+ 860us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.866358 (+ 84us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.870079 (+ 3721us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:03.870112 (+ 33us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:03.870118 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:03.870161 (+ 43us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:03.873748 (+ 3587us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.878171 (+ 4423us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.878178 (+ 7us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.878181 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.878531 (+ 350us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.879320 (+ 789us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.879325 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.879328 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.879636 (+ 308us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:03.879834 (+ 198us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:03.882985 (+ 3151us) server_negotiation.cc:264] Negotiation successful 0927 22:54:03.887218 (+ 4233us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":62} I0927 22:54:03.887804 18454 ts_manager.cc:81] Registered new tserver with Master: 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2:41325) I0927 22:54:03.888494 18769 heartbeater.cc:346] Connected to a master server at 127.17.51.61:11081 I0927 22:54:03.888573 18769 heartbeater.cc:426] Registering TS with master... I0927 22:54:03.888831 18769 heartbeater.cc:475] Master 127.17.51.61:11081 requested a full tablet report, sending... I0927 22:54:03.889873 18389 ts_manager.cc:81] Registered new tserver with Master: 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2:41325) I0927 22:54:03.893791 18774 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:03.852863 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:03.854526 (+ 1663us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:03.854539 (+ 13us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:03.860020 (+ 5481us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:03.860772 (+ 752us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:03.860783 (+ 11us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:03.863726 (+ 2943us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:03.863846 (+ 120us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.876726 (+ 12880us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.876737 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.877352 (+ 615us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.877357 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.878126 (+ 769us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:03.878139 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.879610 (+ 1471us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.879628 (+ 18us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:03.879794 (+ 166us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.880032 (+ 238us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:03.880036 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:03.882268 (+ 2232us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:03.890923 (+ 8655us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.890929 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.890932 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.891351 (+ 419us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.892097 (+ 746us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:03.892101 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:03.892103 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:03.892170 (+ 67us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:03.892872 (+ 702us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:03.892877 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:03.893045 (+ 168us) client_negotiation.cc:695] Sending connection context 0927 22:54:03.893286 (+ 241us) client_negotiation.cc:225] Negotiation successful 0927 22:54:03.893439 (+ 153us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1485,"mutex_wait_us":3860,"thread_start_us":1196,"threads_started":1} I0927 22:54:03.894187 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:03.851430 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.2:44281 0927 22:54:03.851642 (+ 212us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:03.851648 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:03.854587 (+ 2939us) server_negotiation.cc:373] Connection header received 0927 22:54:03.860244 (+ 5657us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:03.860250 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:03.860359 (+ 109us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:03.860439 (+ 80us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:03.876887 (+ 16448us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.877177 (+ 290us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.878277 (+ 1100us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:03.879150 (+ 873us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:03.879247 (+ 97us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:03.882724 (+ 3477us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:03.882772 (+ 48us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:03.882778 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:03.882825 (+ 47us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:03.886491 (+ 3666us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.891596 (+ 5105us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.891600 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.891602 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.891887 (+ 285us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:03.892356 (+ 469us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:03.892360 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:03.892362 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:03.892638 (+ 276us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:03.892815 (+ 177us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:03.893408 (+ 593us) server_negotiation.cc:264] Negotiation successful 0927 22:54:03.893830 (+ 422us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":60} I0927 22:54:03.895006 18768 heartbeater.cc:346] Connected to a master server at 127.17.51.62:11080 I0927 22:54:03.895095 18768 heartbeater.cc:426] Registering TS with master... I0927 22:54:03.895328 18768 heartbeater.cc:475] Master 127.17.51.62:11080 requested a full tablet report, sending... I0927 22:54:03.896405 18324 ts_manager.cc:81] Registered new tserver with Master: 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2:41325) I0927 22:54:03.897720 18324 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.51.2@KRBTEST.COM'} at 127.17.51.2:44281 Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.17.51.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.3.keytab. Entry for principal kudu/127.17.51.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.3.keytab. Entry for principal kudu/127.17.51.3 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.3.keytab. Entry for principal kudu/127.17.51.3 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.3.keytab. I0927 22:54:03.903937 17612 external_mini_cluster.cc:809] Running /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.17.51.3:0 --local_ip_for_outbound_sockets=127.17.51.3 --webserver_interface=127.17.51.3 --webserver_port=0 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.3.keytab --principal=kudu/127.17.51.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --raft_heartbeat_interval_ms=25 --leader_failure_exp_backoff_max_delta_ms=1000 --heartbeat_interval_ms=25 with env {KRB5CCNAME=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0927 22:54:03.924544 18768 heartbeater.cc:467] Master 127.17.51.62:11080 was elected leader, sending a full tablet report... I0927 22:54:03.992403 18501 catalog_manager.cc:1016] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: acquiring CA information for follower catalog manager: success I0927 22:54:03.993710 18501 catalog_manager.cc:1044] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0927 22:54:04.077749 18777 flags.cc:412] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0927 22:54:04.077940 18777 flags.cc:412] Enabled unsafe flag: --principal=kudu/127.17.51.3 W0927 22:54:04.078039 18777 flags.cc:412] Enabled unsafe flag: --never_fsync=true W0927 22:54:04.080348 18777 flags.cc:412] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=1000 W0927 22:54:04.080453 18777 flags.cc:412] Enabled experimental flag: --rpc_trace_negotiation=true W0927 22:54:04.080525 18777 flags.cc:412] Enabled experimental flag: --ipki_server_key_size=1024 W0927 22:54:04.080610 18777 flags.cc:412] Enabled experimental flag: --principal=kudu/127.17.51.3 W0927 22:54:04.080772 18777 flags.cc:412] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.51.3 I0927 22:54:04.085831 18777 tablet_server_main.cc:72] Tablet server non-default flags: --leader_failure_exp_backoff_max_delta_ms=1000 --raft_heartbeat_interval_ms=25 --fs_data_dirs=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/data --fs_wal_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.17.51.3:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/krb5kdc/kudu_127.17.51.3.keytab --principal=kudu/127.17.51.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/data/info.pb --webserver_interface=127.17.51.3 --webserver_port=0 --heartbeat_interval_ms=25 --tserver_master_addrs=127.17.51.62:11080,127.17.51.61:11081,127.17.51.60:11082 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.17.51.3 --log_dir=/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.7.0-cdh5.16.0-SNAPSHOT revision e07a010baa4b0308eb643dc11606b84d5bae6405 build type FASTDEBUG built by jenkins at 27 Sep 2018 15:19:31 PST on quasar-cqqvas-1.vpc.cloudera.com build id 150603 ASAN enabled I0927 22:54:04.087431 18511 catalog_manager.cc:1044] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0927 22:54:04.090632 18777 tablet_server_main.cc:79] Initializing tablet server... I0927 22:54:04.091253 18777 system_ntp.cc:143] NTP initialized. Skew: 500ppm Current error: 113808us Sep 27 22:54:04 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088844, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0927 22:54:04.093094 18777 init.cc:339] Logged in from keytab as kudu/127.17.51.3@KRBTEST.COM (short username kudu) I0927 22:54:04.093901 18777 fs_manager.cc:260] Metadata directory not provided I0927 22:54:04.093971 18777 fs_manager.cc:266] Using write-ahead log directory (fs_wal_dir) as metadata directory I0927 22:54:04.094132 18777 server_base.cc:434] Could not load existing FS layout: Not found: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/data/instance: No such file or directory (error 2) I0927 22:54:04.094241 18777 server_base.cc:435] Attempting to create new FS layout instead I0927 22:54:04.097352 18777 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/data/instance: uuid: "54434d11164442ca96a66cda770a149c" format_stamp: "Formatted at 2018-09-27 22:54:04 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:04.097911 18777 fs_manager.cc:595] Generated new instance metadata in path /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/wal/instance: uuid: "54434d11164442ca96a66cda770a149c" format_stamp: "Formatted at 2018-09-27 22:54:04 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:04.102937 18777 fs_manager.cc:495] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s I0927 22:54:04.103941 18777 env_posix.cc:1634] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0927 22:54:04.104194 18777 file_cache.cc:470] Constructed file cache lbm with capacity 419430 I0927 22:54:04.106884 18777 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0927 22:54:04.106947 18777 fs_manager.cc:428] Opened local filesystem: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/data,/tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/wal uuid: "54434d11164442ca96a66cda770a149c" format_stamp: "Formatted at 2018-09-27 22:54:04 on dist-test-slave-dist-test-slave-xppb" I0927 22:54:04.107112 18777 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0927 22:54:04.147450 18777 env_posix.cc:1634] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0927 22:54:04.148252 18777 ts_tablet_manager.cc:299] Loaded tablet metadata (0 live tablets) I0927 22:54:04.148558 18777 tablet_server_main.cc:84] Starting tablet server... I0927 22:54:04.176674 18777 rpc_server.cc:201] RPC server started. Bound to: 127.17.51.3:44871 I0927 22:54:04.177093 18777 webserver.cc:173] Starting webserver on 127.17.51.3:0 I0927 22:54:04.177161 18777 webserver.cc:184] Document root disabled I0927 22:54:04.177795 18777 webserver.cc:311] Webserver started. Bound to: http://127.17.51.3:33313/ I0927 22:54:04.178411 18777 server_base.cc:607] Dumped server information to /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/data/info.pb I0927 22:54:04.186965 17612 external_mini_cluster.cc:871] Started /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver as pid 18777 I0927 22:54:04.205427 18901 process_memory.cc:182] Process hard memory limit is 11.744253 GB I0927 22:54:04.205617 18901 process_memory.cc:184] Process soft memory limit is 9.395403 GB I0927 22:54:04.205631 18777 tablet_server_main.cc:87] Tablet server successfully started. I0927 22:54:04.205674 18901 process_memory.cc:187] Process memory pressure threshold is 7.046552 GB Sep 27 22:54:04 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM Sep 27 22:54:04 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088844, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.3@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM Sep 27 22:54:04 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088844, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.3@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM I0927 22:54:04.225513 18903 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.187809 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:04.206470 (+ 18661us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.206490 (+ 20us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.206616 (+ 126us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.207745 (+ 1129us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.207750 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.208811 (+ 1061us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.208939 (+ 128us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.209077 (+ 138us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.209088 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.209676 (+ 588us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.209681 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.210365 (+ 684us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.210376 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.211955 (+ 1579us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.211963 (+ 8us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.212116 (+ 153us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.212311 (+ 195us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.212315 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.215894 (+ 3579us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.221591 (+ 5697us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.221596 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.221600 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.222012 (+ 412us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.222743 (+ 731us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.222746 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.222749 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.222802 (+ 53us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.223527 (+ 725us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.223533 (+ 6us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.223685 (+ 152us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.223915 (+ 230us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.224241 (+ 326us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":18466,"thread_start_us":19637,"threads_started":1} Sep 27 22:54:04 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088844, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.51.3@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM I0927 22:54:04.227804 18905 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.188437 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42506 0927 22:54:04.207092 (+ 18655us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.207098 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.207118 (+ 20us) server_negotiation.cc:373] Connection header received 0927 22:54:04.207187 (+ 69us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.207192 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.207285 (+ 93us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.207373 (+ 88us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.209244 (+ 1871us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.209530 (+ 286us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.210510 (+ 980us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.211448 (+ 938us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.211545 (+ 97us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.216415 (+ 4870us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.216551 (+ 136us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.216561 (+ 10us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.216600 (+ 39us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.221216 (+ 4616us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.222247 (+ 1031us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.222251 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.222252 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.222502 (+ 250us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.223013 (+ 511us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.223016 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.223018 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.223259 (+ 241us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.223620 (+ 361us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.226986 (+ 3366us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.227475 (+ 489us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":2838,"server-negotiator.queue_time_us":18028,"thread_start_us":21531,"threads_started":1} I0927 22:54:04.232983 18489 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.188559 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.3:34323 0927 22:54:04.188766 (+ 207us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.188772 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.207677 (+ 18905us) server_negotiation.cc:373] Connection header received 0927 22:54:04.212449 (+ 4772us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.212455 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.212549 (+ 94us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.212613 (+ 64us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.214638 (+ 2025us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.214872 (+ 234us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.216046 (+ 1174us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.216948 (+ 902us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.217042 (+ 94us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.223329 (+ 6287us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.223363 (+ 34us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.223368 (+ 5us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.223407 (+ 39us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.226723 (+ 3316us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.228933 (+ 2210us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.228939 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.228942 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.229317 (+ 375us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.230159 (+ 842us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.230165 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.230168 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.230410 (+ 242us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.230604 (+ 194us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.231838 (+ 1234us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.232560 (+ 722us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":52} I0927 22:54:04.234402 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.180015 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.3:43083 0927 22:54:04.180265 (+ 250us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.180271 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.186638 (+ 6367us) server_negotiation.cc:373] Connection header received 0927 22:54:04.212993 (+ 26355us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.212999 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.213115 (+ 116us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.213192 (+ 77us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.214659 (+ 1467us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.214922 (+ 263us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.221552 (+ 6630us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.222416 (+ 864us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.222504 (+ 88us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.225830 (+ 3326us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.225870 (+ 40us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.225876 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.225918 (+ 42us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.229477 (+ 3559us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.230723 (+ 1246us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.230730 (+ 7us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.230733 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.231096 (+ 363us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.231914 (+ 818us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.231919 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.231922 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.232228 (+ 306us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.232449 (+ 221us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.233366 (+ 917us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.234026 (+ 660us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":78} I0927 22:54:04.234488 18904 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.188018 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:04.207607 (+ 19589us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.207621 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.212318 (+ 4697us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.212782 (+ 464us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.212788 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.213747 (+ 959us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.214170 (+ 423us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.214501 (+ 331us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.214512 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.215032 (+ 520us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.215037 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.215887 (+ 850us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.215900 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.217460 (+ 1560us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.217467 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.217638 (+ 171us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.217927 (+ 289us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.217931 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.222969 (+ 5038us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.226992 (+ 4023us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.227000 (+ 8us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.227008 (+ 8us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.228581 (+ 1573us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.229680 (+ 1099us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.229686 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.229689 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.229819 (+ 130us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.230829 (+ 1010us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.230837 (+ 8us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.231162 (+ 325us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.233198 (+ 2036us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.233468 (+ 270us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":19451,"mutex_wait_us":1185,"spinlock_wait_cycles":58752,"thread_start_us":19322,"threads_started":1} I0927 22:54:04.236075 18900 heartbeater.cc:346] Connected to a master server at 127.17.51.60:11082 I0927 22:54:04.236259 18900 heartbeater.cc:426] Registering TS with master... I0927 22:54:04.236768 18899 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.180402 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.186520 (+ 6118us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.186559 (+ 39us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.212149 (+ 25590us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.213434 (+ 1285us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.213441 (+ 7us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.213852 (+ 411us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.214178 (+ 326us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.214507 (+ 329us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.214519 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.220366 (+ 5847us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.220371 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.221396 (+ 1025us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.221409 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.222822 (+ 1413us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.222828 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.222967 (+ 139us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.223210 (+ 243us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.223214 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.225438 (+ 2224us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.229936 (+ 4498us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.229941 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.229945 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.230409 (+ 464us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.231514 (+ 1105us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.231519 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.231522 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.231609 (+ 87us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.232717 (+ 1108us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.232723 (+ 6us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.232883 (+ 160us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.234867 (+ 1984us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.235853 (+ 986us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":5586,"mutex_wait_us":820,"spinlock_wait_cycles":41984,"thread_start_us":5100,"threads_started":1} I0927 22:54:04.236876 18900 heartbeater.cc:475] Master 127.17.51.60:11082 requested a full tablet report, sending... I0927 22:54:04.238487 18453 ts_manager.cc:81] Registered new tserver with Master: 54434d11164442ca96a66cda770a149c (127.17.51.3:44871) I0927 22:54:04.240767 18897 heartbeater.cc:346] Connected to a master server at 127.17.51.62:11080 I0927 22:54:04.240959 18897 heartbeater.cc:426] Registering TS with master... I0927 22:54:04.241196 18897 heartbeater.cc:475] Master 127.17.51.62:11080 requested a full tablet report, sending... I0927 22:54:04.242033 18642 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.186901 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.3:54491 0927 22:54:04.187164 (+ 263us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.187170 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.206922 (+ 19752us) server_negotiation.cc:373] Connection header received 0927 22:54:04.212344 (+ 5422us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.212350 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.212450 (+ 100us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.212597 (+ 147us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.214638 (+ 2041us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.214902 (+ 264us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.219664 (+ 4762us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.220270 (+ 606us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.220371 (+ 101us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.228250 (+ 7879us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.228284 (+ 34us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.228289 (+ 5us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.228333 (+ 44us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.234260 (+ 5927us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.236217 (+ 1957us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.236224 (+ 7us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.236227 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.236634 (+ 407us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.237792 (+ 1158us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.237799 (+ 7us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.237802 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.238048 (+ 246us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.238285 (+ 237us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.239204 (+ 919us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.239874 (+ 670us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":82} I0927 22:54:04.242766 18902 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.187307 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:04.206844 (+ 19537us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.206864 (+ 20us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.212150 (+ 5286us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.212766 (+ 616us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.212774 (+ 8us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.213747 (+ 973us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.214170 (+ 423us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.214438 (+ 268us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.214462 (+ 24us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.217001 (+ 2539us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.217006 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.219535 (+ 2529us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.219547 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.220486 (+ 939us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.220493 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.220622 (+ 129us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.220822 (+ 200us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.220826 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.227936 (+ 7110us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.234632 (+ 6696us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.234638 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.234641 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.235113 (+ 472us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.237363 (+ 2250us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.237368 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.237372 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.237454 (+ 82us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.238494 (+ 1040us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.238499 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.238669 (+ 170us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.242251 (+ 3582us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.242415 (+ 164us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":19323,"mutex_wait_us":4611,"thread_start_us":19113,"threads_started":1} I0927 22:54:04.242755 18324 ts_manager.cc:81] Registered new tserver with Master: 54434d11164442ca96a66cda770a149c (127.17.51.3:44871) I0927 22:54:04.243928 18898 heartbeater.cc:346] Connected to a master server at 127.17.51.61:11081 I0927 22:54:04.244004 18898 heartbeater.cc:426] Registering TS with master... Sep 27 22:54:04 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM I0927 22:54:04.244045 18324 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.51.3@KRBTEST.COM'} at 127.17.51.3:43083 I0927 22:54:04.244279 18898 heartbeater.cc:475] Master 127.17.51.61:11081 requested a full tablet report, sending... I0927 22:54:04.246592 18388 ts_manager.cc:81] Registered new tserver with Master: 54434d11164442ca96a66cda770a149c (127.17.51.3:44871) I0927 22:54:04.252205 18903 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.231077 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:04.235581 (+ 4504us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.235597 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.235721 (+ 124us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.236149 (+ 428us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.236154 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.237298 (+ 1144us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.237385 (+ 87us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.237491 (+ 106us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.237504 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.238168 (+ 664us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.238174 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.238839 (+ 665us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.238853 (+ 14us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.240217 (+ 1364us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.240225 (+ 8us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.240367 (+ 142us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.240543 (+ 176us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.240547 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.245035 (+ 4488us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.249848 (+ 4813us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.249852 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.249855 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.250178 (+ 323us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.250791 (+ 613us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.250794 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.250796 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.250848 (+ 52us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.251445 (+ 597us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.251449 (+ 4us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.251561 (+ 112us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.251712 (+ 151us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.251940 (+ 228us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":4351} I0927 22:54:04.252593 18906 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.229914 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51212 0927 22:54:04.235622 (+ 5708us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.235627 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.235649 (+ 22us) server_negotiation.cc:373] Connection header received 0927 22:54:04.235886 (+ 237us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.235892 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.235979 (+ 87us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.236090 (+ 111us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.237661 (+ 1571us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.237981 (+ 320us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.239001 (+ 1020us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.239971 (+ 970us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.240154 (+ 183us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.245455 (+ 5301us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.245488 (+ 33us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.245494 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.245529 (+ 35us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.249602 (+ 4073us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.250374 (+ 772us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.250378 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.250380 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.250599 (+ 219us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.251013 (+ 414us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.251016 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.251018 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.251219 (+ 201us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.251452 (+ 233us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.251830 (+ 378us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.252232 (+ 402us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":5532,"thread_start_us":5400,"threads_started":1} Sep 27 22:54:04 dist-test-slave-dist-test-slave-xppb krb5kdc[18287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM I0927 22:54:04.267140 18903 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.254307 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.254516 (+ 209us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.254528 (+ 12us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.254615 (+ 87us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.254910 (+ 295us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.254914 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.255641 (+ 727us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.255698 (+ 57us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.255777 (+ 79us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.255785 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.257025 (+ 1240us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.257028 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.257603 (+ 575us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.257613 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.258685 (+ 1072us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.258691 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.258826 (+ 135us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.258967 (+ 141us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.258969 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.261129 (+ 2160us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.264103 (+ 2974us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.264107 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.264109 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.264623 (+ 514us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.265578 (+ 955us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.265582 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.265584 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.265636 (+ 52us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.266363 (+ 727us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.266367 (+ 4us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.266470 (+ 103us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.266642 (+ 172us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.266850 (+ 208us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":62} I0927 22:54:04.267498 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.254396 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40420 0927 22:54:04.254584 (+ 188us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.254588 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.254606 (+ 18us) server_negotiation.cc:373] Connection header received 0927 22:54:04.254715 (+ 109us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.254719 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.254785 (+ 66us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.254870 (+ 85us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.255893 (+ 1023us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.256898 (+ 1005us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.257789 (+ 891us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.258527 (+ 738us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.258635 (+ 108us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.261403 (+ 2768us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.261428 (+ 25us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.261431 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.261463 (+ 32us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.263851 (+ 2388us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.264892 (+ 1041us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.264898 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.264901 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.265323 (+ 422us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.265866 (+ 543us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.265870 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.265872 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.266098 (+ 226us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.266407 (+ 309us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.266742 (+ 335us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.267182 (+ 440us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":72} I0927 22:54:04.271728 18897 heartbeater.cc:467] Master 127.17.51.62:11080 was elected leader, sending a full tablet report... I0927 22:54:04.285418 18903 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.271180 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:04.271387 (+ 207us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.271404 (+ 17us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.271497 (+ 93us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.271835 (+ 338us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.271840 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.276905 (+ 5065us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.276968 (+ 63us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.277021 (+ 53us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.277029 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.277499 (+ 470us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.277502 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.278034 (+ 532us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.278044 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.278986 (+ 942us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.278990 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.279100 (+ 110us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.279232 (+ 132us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.279234 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.280071 (+ 837us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.283046 (+ 2975us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.283050 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.283052 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.283373 (+ 321us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.283995 (+ 622us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.283998 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.284000 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.284050 (+ 50us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.284670 (+ 620us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.284673 (+ 3us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.284767 (+ 94us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.284912 (+ 145us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.285146 (+ 234us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":68} I0927 22:54:04.285845 18905 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.271239 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42512 0927 22:54:04.271462 (+ 223us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.271468 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.271491 (+ 23us) server_negotiation.cc:373] Connection header received 0927 22:54:04.271616 (+ 125us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.271621 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.271704 (+ 83us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.271797 (+ 93us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.277178 (+ 5381us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.277373 (+ 195us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.278191 (+ 818us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.278850 (+ 659us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.278935 (+ 85us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.280328 (+ 1393us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.280355 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.280359 (+ 4us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.280393 (+ 34us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.282830 (+ 2437us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.283576 (+ 746us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.283580 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.283581 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.283815 (+ 234us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.284236 (+ 421us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.284239 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.284241 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.284413 (+ 172us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.284720 (+ 307us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.285026 (+ 306us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.285519 (+ 493us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":73} I0927 22:54:04.287030 17612 external_mini_cluster.cc:499] 3 TS(s) registered with all masters I0927 22:54:04.309767 18911 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.291730 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:04.293917 (+ 2187us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.293932 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.294145 (+ 213us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.294548 (+ 403us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.294553 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.296071 (+ 1518us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.296210 (+ 139us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.296314 (+ 104us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.296326 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.296986 (+ 660us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.296991 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.298892 (+ 1901us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.298906 (+ 14us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.300382 (+ 1476us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.300388 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.301339 (+ 951us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.301543 (+ 204us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.301547 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.303025 (+ 1478us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.306613 (+ 3588us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.306618 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.306622 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.306949 (+ 327us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.307809 (+ 860us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.307814 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.307818 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.307885 (+ 67us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.308703 (+ 818us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.308708 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.308853 (+ 145us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.309083 (+ 230us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.309257 (+ 174us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1972,"mutex_wait_us":3,"thread_start_us":2014,"threads_started":1} I0927 22:54:04.311679 18906 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.291305 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51220 0927 22:54:04.291513 (+ 208us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.291520 (+ 7us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.293998 (+ 2478us) server_negotiation.cc:373] Connection header received 0927 22:54:04.294271 (+ 273us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.294278 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.294386 (+ 108us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.294495 (+ 109us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.296486 (+ 1991us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.296857 (+ 371us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.299073 (+ 2216us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.299921 (+ 848us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.300011 (+ 90us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.303335 (+ 3324us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.303366 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.303371 (+ 5us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.303412 (+ 41us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.306304 (+ 2892us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.307193 (+ 889us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.307199 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.307202 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.307540 (+ 338us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.308150 (+ 610us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.308155 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.308158 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.308408 (+ 250us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.308774 (+ 366us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.309285 (+ 511us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.309916 (+ 631us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":65} I0927 22:54:04.314613 18912 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.291000 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.293363 (+ 2363us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.293379 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.293470 (+ 91us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.294672 (+ 1202us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.294677 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.296457 (+ 1780us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.296581 (+ 124us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.296654 (+ 73us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.296662 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.297162 (+ 500us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.297168 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.297951 (+ 783us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.297963 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.304009 (+ 6046us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.304014 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.304118 (+ 104us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.304253 (+ 135us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.304255 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.305510 (+ 1255us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.309563 (+ 4053us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.309568 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.309572 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.311075 (+ 1503us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.312446 (+ 1371us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.312450 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.312452 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.312611 (+ 159us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.313503 (+ 892us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.313508 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.313646 (+ 138us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.313991 (+ 345us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.314245 (+ 254us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":183,"thread_start_us":2597,"threads_started":1} I0927 22:54:04.314997 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.290970 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40424 0927 22:54:04.291192 (+ 222us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.294252 (+ 3060us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.294271 (+ 19us) server_negotiation.cc:373] Connection header received 0927 22:54:04.294316 (+ 45us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.294320 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.294403 (+ 83us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.294495 (+ 92us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.296787 (+ 2292us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.297017 (+ 230us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.298116 (+ 1099us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.299564 (+ 1448us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.299658 (+ 94us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.305852 (+ 6194us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.305884 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.305890 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.305931 (+ 41us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.309148 (+ 3217us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.311463 (+ 2315us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.311469 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.311472 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.311845 (+ 373us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.312948 (+ 1103us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.312954 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.312957 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.313253 (+ 296us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.313608 (+ 355us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.314131 (+ 523us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.314659 (+ 528us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":75} I0927 22:54:04.315750 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.294312 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:04.295245 (+ 933us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.295256 (+ 11us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.295376 (+ 120us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.295699 (+ 323us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.295704 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.298134 (+ 2430us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.298221 (+ 87us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.298285 (+ 64us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.298293 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.298801 (+ 508us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.298806 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.299455 (+ 649us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.299468 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.304444 (+ 4976us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.304450 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.304616 (+ 166us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.305011 (+ 395us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.305014 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.306376 (+ 1362us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.312142 (+ 5766us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.312146 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.312148 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.312568 (+ 420us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.313644 (+ 1076us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.313649 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.313653 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.313757 (+ 104us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.314557 (+ 800us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.314562 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.314877 (+ 315us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.315297 (+ 420us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.315499 (+ 202us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":1092,"thread_start_us":548,"threads_started":1} I0927 22:54:04.316524 18905 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.292128 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42518 0927 22:54:04.294223 (+ 2095us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.294227 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.295321 (+ 1094us) server_negotiation.cc:373] Connection header received 0927 22:54:04.295486 (+ 165us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.295492 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.295566 (+ 74us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.295648 (+ 82us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.298413 (+ 2765us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.298661 (+ 248us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.299614 (+ 953us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.304283 (+ 4669us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.304392 (+ 109us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.307315 (+ 2923us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.307341 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.307344 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.307378 (+ 34us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.311755 (+ 4377us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.312878 (+ 1123us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.312884 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.312887 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.313315 (+ 428us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.314000 (+ 685us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.314006 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.314009 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.314306 (+ 297us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.314602 (+ 296us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.315470 (+ 868us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.316124 (+ 654us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1934} I0927 22:54:04.325791 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.321680 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.321917 (+ 237us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.321931 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.322023 (+ 92us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.322390 (+ 367us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.322394 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.322404 (+ 10us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.322473 (+ 69us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.322523 (+ 50us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.322530 (+ 7us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.322938 (+ 408us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.322941 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.323608 (+ 667us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.323618 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.324542 (+ 924us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.324549 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.324659 (+ 110us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.324726 (+ 67us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.325424 (+ 698us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.325439 (+ 15us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.325565 (+ 126us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.325607 (+ 42us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":61} I0927 22:54:04.325844 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.321762 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40430 0927 22:54:04.321948 (+ 186us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.321951 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.321967 (+ 16us) server_negotiation.cc:373] Connection header received 0927 22:54:04.322173 (+ 206us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.322177 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.322264 (+ 87us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.322348 (+ 84us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.322629 (+ 281us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.322826 (+ 197us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.323750 (+ 924us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.324362 (+ 612us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.324447 (+ 85us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.324997 (+ 550us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.325219 (+ 222us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.325354 (+ 135us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.325629 (+ 275us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.325660 (+ 31us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":61} I0927 22:54:04.326715 18324 catalog_manager.cc:1294] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40430: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "val" type: INT32 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0927 22:54:04.334163 18324 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0927 22:54:04.350045 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.343925 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.344903 (+ 978us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.344919 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.345047 (+ 128us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.345404 (+ 357us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.345409 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.345423 (+ 14us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.345500 (+ 77us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.345604 (+ 104us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.345617 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.346171 (+ 554us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.346178 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.347139 (+ 961us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.347152 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.348331 (+ 1179us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.348338 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.348504 (+ 166us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.348571 (+ 67us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.349464 (+ 893us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.349476 (+ 12us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.349629 (+ 153us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.349675 (+ 46us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":813} I0927 22:54:04.351790 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.344218 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40432 0927 22:54:04.344656 (+ 438us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.344662 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.344988 (+ 326us) server_negotiation.cc:373] Connection header received 0927 22:54:04.345186 (+ 198us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.345191 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.345275 (+ 84us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.345365 (+ 90us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.345753 (+ 388us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.345977 (+ 224us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.347295 (+ 1318us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.348134 (+ 839us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.348240 (+ 106us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.348940 (+ 700us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.349127 (+ 187us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.349299 (+ 172us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.351382 (+ 2083us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.351430 (+ 48us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":304} I0927 22:54:04.370498 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.355667 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.355944 (+ 277us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.355960 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.356088 (+ 128us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.365804 (+ 9716us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.365810 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.365825 (+ 15us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.365913 (+ 88us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.365982 (+ 69us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.365993 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.366882 (+ 889us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.366886 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.367605 (+ 719us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.367617 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.368933 (+ 1316us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.368939 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.369090 (+ 151us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.369154 (+ 64us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.369947 (+ 793us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.369959 (+ 12us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.370124 (+ 165us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.370171 (+ 47us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":128} I0927 22:54:04.372588 18928 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.358444 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:32812 0927 22:54:04.364292 (+ 5848us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.364300 (+ 8us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.364688 (+ 388us) server_negotiation.cc:373] Connection header received 0927 22:54:04.364796 (+ 108us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.364804 (+ 8us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.365003 (+ 199us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.365166 (+ 163us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:54:04.365872 (+ 706us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.366176 (+ 304us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.368429 (+ 2253us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.370271 (+ 1842us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.370369 (+ 98us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.370524 (+ 155us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.371442 (+ 918us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.371506 (+ 64us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5667,"thread_start_us":5662,"threads_started":1} I0927 22:54:04.372658 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.356249 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40434 0927 22:54:04.365376 (+ 9127us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.365381 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.365398 (+ 17us) server_negotiation.cc:373] Connection header received 0927 22:54:04.365448 (+ 50us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.365453 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.365555 (+ 102us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.365652 (+ 97us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.366385 (+ 733us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.366721 (+ 336us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.367754 (+ 1033us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.368677 (+ 923us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.368773 (+ 96us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.369495 (+ 722us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.369653 (+ 158us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.369788 (+ 135us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.370987 (+ 1199us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.371027 (+ 40us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":9019} I0927 22:54:04.372694 18927 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.358111 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.1:40273 0927 22:54:04.364427 (+ 6316us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.364441 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.364580 (+ 139us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.365376 (+ 796us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.365380 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.365412 (+ 32us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:54:04.365546 (+ 134us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.365718 (+ 172us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.365728 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.366323 (+ 595us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.366328 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.368260 (+ 1932us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.368272 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.370681 (+ 2409us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.370688 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.370833 (+ 145us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.370890 (+ 57us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.371041 (+ 151us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.371108 (+ 67us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":6170,"mutex_wait_us":24,"thread_start_us":5887,"threads_started":1} I0927 22:54:04.373528 18924 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.356984 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.2:41325 0927 22:54:04.361680 (+ 4696us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.361693 (+ 13us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.361781 (+ 88us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.362924 (+ 1143us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.362930 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.362962 (+ 32us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:54:04.363936 (+ 974us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.365882 (+ 1946us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.365894 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.366495 (+ 601us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.366502 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.368316 (+ 1814us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.368331 (+ 15us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.370935 (+ 2604us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.370941 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.371120 (+ 179us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.371175 (+ 55us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.373113 (+ 1938us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.373153 (+ 40us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":4600,"mutex_wait_us":23,"thread_start_us":7313,"threads_started":1} I0927 22:54:04.374325 18923 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.356455 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.3:44871 0927 22:54:04.365634 (+ 9179us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.365650 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.365744 (+ 94us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.366495 (+ 751us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.366501 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.366533 (+ 32us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:54:04.366862 (+ 329us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.367031 (+ 169us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.367043 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.367846 (+ 803us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.367856 (+ 10us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.369575 (+ 1719us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.369588 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.373406 (+ 3818us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.373412 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.373750 (+ 338us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.373805 (+ 55us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.373974 (+ 169us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.374010 (+ 36us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":921,"threads_started":1} I0927 22:54:04.374469 18926 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.357378 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40738 0927 22:54:04.362515 (+ 5137us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.362520 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.362542 (+ 22us) server_negotiation.cc:373] Connection header received 0927 22:54:04.362604 (+ 62us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.362609 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.362780 (+ 171us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.362905 (+ 125us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:54:04.366037 (+ 3132us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.366347 (+ 310us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.368535 (+ 2188us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.370530 (+ 1995us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.370629 (+ 99us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.370773 (+ 144us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.374008 (+ 3235us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.374142 (+ 134us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5006,"thread_start_us":4876,"threads_started":1} I0927 22:54:04.374505 18925 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.356897 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42128 0927 22:54:04.362158 (+ 5261us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.362164 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.365850 (+ 3686us) server_negotiation.cc:373] Connection header received 0927 22:54:04.365942 (+ 92us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.365949 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.366157 (+ 208us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.366277 (+ 120us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:54:04.367210 (+ 933us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.367492 (+ 282us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.369730 (+ 2238us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.372580 (+ 2850us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.373089 (+ 509us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.373248 (+ 159us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.374137 (+ 889us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.374200 (+ 63us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5100,"thread_start_us":4984,"threads_started":1} I0927 22:54:04.376092 18574 tablet_service.cc:758] Processing CreateTablet for tablet ef02af55cbde4ce3a33f3f4cb329f1b5 (table=test-table [id=40586b0786544a20b9f645e03b20463b]), partition=RANGE (key) PARTITION UNBOUNDED I0927 22:54:04.376963 18574 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet ef02af55cbde4ce3a33f3f4cb329f1b5. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:54:04.377823 18834 tablet_service.cc:758] Processing CreateTablet for tablet ef02af55cbde4ce3a33f3f4cb329f1b5 (table=test-table [id=40586b0786544a20b9f645e03b20463b]), partition=RANGE (key) PARTITION UNBOUNDED I0927 22:54:04.378608 18834 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet ef02af55cbde4ce3a33f3f4cb329f1b5. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:54:04.380249 18705 tablet_service.cc:758] Processing CreateTablet for tablet ef02af55cbde4ce3a33f3f4cb329f1b5 (table=test-table [id=40586b0786544a20b9f645e03b20463b]), partition=RANGE (key) PARTITION UNBOUNDED I0927 22:54:04.382901 18705 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet ef02af55cbde4ce3a33f3f4cb329f1b5. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:54:04.385093 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.377850 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.379639 (+ 1789us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.379655 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.379756 (+ 101us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.380287 (+ 531us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.380292 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.380306 (+ 14us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.380379 (+ 73us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.380444 (+ 65us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.380454 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.381094 (+ 640us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.381099 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.382077 (+ 978us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.382091 (+ 14us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.383333 (+ 1242us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.383340 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.383472 (+ 132us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.383542 (+ 70us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.384449 (+ 907us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.384460 (+ 11us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.384713 (+ 253us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.384757 (+ 44us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1606} I0927 22:54:04.385833 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.378146 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40442 0927 22:54:04.379971 (+ 1825us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.379976 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.379994 (+ 18us) server_negotiation.cc:373] Connection header received 0927 22:54:04.380040 (+ 46us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.380045 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.380149 (+ 104us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.380608 (+ 459us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.380745 (+ 137us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.380933 (+ 188us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.382260 (+ 1327us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.383172 (+ 912us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.383768 (+ 596us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.383984 (+ 216us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.384152 (+ 168us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.385408 (+ 1256us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.385513 (+ 105us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.385546 (+ 33us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1695} I0927 22:54:04.394361 18834 ts_tablet_manager.cc:1080] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Registered tablet (data state: TABLET_DATA_READY) I0927 22:54:04.395017 18929 ts_tablet_manager.cc:938] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Bootstrapping tablet I0927 22:54:04.395033 18574 ts_tablet_manager.cc:1080] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Registered tablet (data state: TABLET_DATA_READY) I0927 22:54:04.395428 18929 tablet_bootstrap.cc:436] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Bootstrap starting. I0927 22:54:04.396752 18930 ts_tablet_manager.cc:938] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Bootstrapping tablet I0927 22:54:04.397033 18930 tablet_bootstrap.cc:436] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Bootstrap starting. I0927 22:54:04.397842 18705 ts_tablet_manager.cc:1080] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: Registered tablet (data state: TABLET_DATA_READY) I0927 22:54:04.399595 18931 ts_tablet_manager.cc:938] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: Bootstrapping tablet I0927 22:54:04.399844 18931 tablet_bootstrap.cc:436] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: Bootstrap starting. I0927 22:54:04.400074 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.389533 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.390164 (+ 631us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.390179 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.390293 (+ 114us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.392554 (+ 2261us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.392560 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.392574 (+ 14us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.392654 (+ 80us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.392719 (+ 65us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.392729 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.393448 (+ 719us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.393453 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.394449 (+ 996us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.394461 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.397596 (+ 3135us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.397604 (+ 8us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.397765 (+ 161us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.397834 (+ 69us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.399489 (+ 1655us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.399502 (+ 13us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.399680 (+ 178us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.399732 (+ 52us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":460} I0927 22:54:04.400769 18929 tablet_bootstrap.cc:581] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: No blocks or log segments found. Creating new log. I0927 22:54:04.401103 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.389860 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40444 0927 22:54:04.390046 (+ 186us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.390051 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.390239 (+ 188us) server_negotiation.cc:373] Connection header received 0927 22:54:04.392317 (+ 2078us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.392323 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.392408 (+ 85us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.392829 (+ 421us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.392957 (+ 128us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.393279 (+ 322us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.394641 (+ 1362us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.397416 (+ 2775us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.398051 (+ 635us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.398295 (+ 244us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.398444 (+ 149us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.400727 (+ 2283us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.400850 (+ 123us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.400891 (+ 41us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":63} I0927 22:54:04.401839 18931 tablet_bootstrap.cc:581] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: No blocks or log segments found. Creating new log. I0927 22:54:04.401849 18929 log.cc:520] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Log is configured to *not* fsync() on all Append() calls I0927 22:54:04.402823 18931 log.cc:520] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: Log is configured to *not* fsync() on all Append() calls I0927 22:54:04.403071 18930 tablet_bootstrap.cc:581] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: No blocks or log segments found. Creating new log. I0927 22:54:04.404018 18930 log.cc:520] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Log is configured to *not* fsync() on all Append() calls I0927 22:54:04.407322 18929 tablet_bootstrap.cc:436] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: No bootstrap required, opened a new log I0927 22:54:04.407534 18929 ts_tablet_manager.cc:955] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s I0927 22:54:04.413084 18931 tablet_bootstrap.cc:436] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: No bootstrap required, opened a new log I0927 22:54:04.413311 18931 ts_tablet_manager.cc:955] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s I0927 22:54:04.415267 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.406891 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.409815 (+ 2924us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.409832 (+ 17us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.409940 (+ 108us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.410509 (+ 569us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.410514 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.410527 (+ 13us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.410611 (+ 84us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.410692 (+ 81us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.410702 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.411358 (+ 656us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.411364 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.412104 (+ 740us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.412118 (+ 14us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.413490 (+ 1372us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.413498 (+ 8us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.413663 (+ 165us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.413730 (+ 67us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.414731 (+ 1001us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.414743 (+ 12us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.414886 (+ 143us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.414930 (+ 44us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":2752} I0927 22:54:04.416250 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.407209 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40446 0927 22:54:04.410181 (+ 2972us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.410186 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.410205 (+ 19us) server_negotiation.cc:373] Connection header received 0927 22:54:04.410260 (+ 55us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.410266 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.410366 (+ 100us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.410806 (+ 440us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.410952 (+ 146us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.411203 (+ 251us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.412290 (+ 1087us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.413259 (+ 969us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.414011 (+ 752us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.414269 (+ 258us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.414413 (+ 144us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.415769 (+ 1356us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.415921 (+ 152us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.415963 (+ 42us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":2816} I0927 22:54:04.418306 18930 tablet_bootstrap.cc:436] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: No bootstrap required, opened a new log I0927 22:54:04.418542 18930 ts_tablet_manager.cc:955] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.013s I0927 22:54:04.420399 18929 raft_consensus.cc:304] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.420753 18929 raft_consensus.cc:330] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:54:04.420828 18929 raft_consensus.cc:605] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54434d11164442ca96a66cda770a149c, State: Initialized, Role: FOLLOWER I0927 22:54:04.421248 18929 consensus_queue.cc:226] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.426856 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.421248 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.422017 (+ 769us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.422031 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.422133 (+ 102us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.422589 (+ 456us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.422593 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.422605 (+ 12us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.422668 (+ 63us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.422729 (+ 61us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.422737 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.423347 (+ 610us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.423352 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.424180 (+ 828us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.424192 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.425344 (+ 1152us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.425350 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.425468 (+ 118us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.425529 (+ 61us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.426415 (+ 886us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.426428 (+ 13us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.426540 (+ 112us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.426589 (+ 49us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":638} I0927 22:54:04.427572 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.421758 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40448 0927 22:54:04.422316 (+ 558us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.422320 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.422336 (+ 16us) server_negotiation.cc:373] Connection header received 0927 22:54:04.422379 (+ 43us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.422383 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.422453 (+ 70us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.422826 (+ 373us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.422940 (+ 114us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.423157 (+ 217us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.424367 (+ 1210us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.425194 (+ 827us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.425772 (+ 578us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.425966 (+ 194us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.426132 (+ 166us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.427165 (+ 1033us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.427294 (+ 129us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.427335 (+ 41us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":449} I0927 22:54:04.429595 18929 ts_tablet_manager.cc:983] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Time spent starting tablet: real 0.022s user 0.012s sys 0.004s I0927 22:54:04.431169 18935 raft_consensus.cc:436] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0927 22:54:04.431303 18935 raft_consensus.cc:2700] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Snoozing failure detection for 0.077s (starting election) I0927 22:54:04.431443 18935 raft_consensus.cc:458] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.432647 18935 leader_election.cc:231] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 00f349d4d9e84c4596252016e4bfc620 I0927 22:54:04.438618 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.433115 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.433814 (+ 699us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.433825 (+ 11us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.433932 (+ 107us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.434231 (+ 299us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.434237 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.434250 (+ 13us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.434318 (+ 68us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.434396 (+ 78us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.434406 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.434906 (+ 500us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.434911 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.435514 (+ 603us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.435551 (+ 37us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.437287 (+ 1736us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.437294 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.437412 (+ 118us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.437493 (+ 81us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.438117 (+ 624us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.438127 (+ 10us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.438262 (+ 135us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.438304 (+ 42us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":587} I0927 22:54:04.439431 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.433402 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40450 0927 22:54:04.433593 (+ 191us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.433598 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.433889 (+ 291us) server_negotiation.cc:373] Connection header received 0927 22:54:04.434040 (+ 151us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.434045 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.434123 (+ 78us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.434202 (+ 79us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.434546 (+ 344us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.434765 (+ 219us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.435693 (+ 928us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.437142 (+ 1449us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.437238 (+ 96us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.437769 (+ 531us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.437876 (+ 107us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.438013 (+ 137us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.438372 (+ 359us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.438413 (+ 41us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":61} I0927 22:54:04.441023 18902 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.433147 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.1:40273 0927 22:54:04.433493 (+ 346us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.433509 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.433604 (+ 95us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.434416 (+ 812us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.434421 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.434450 (+ 29us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:54:04.434515 (+ 65us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.434625 (+ 110us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.434636 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.435164 (+ 528us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.435168 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.436668 (+ 1500us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.436680 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.440254 (+ 3574us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.440260 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.440419 (+ 159us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.440496 (+ 77us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.440670 (+ 174us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.440711 (+ 41us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":215} I0927 22:54:04.441773 18928 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.433782 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.3:58945 0927 22:54:04.433991 (+ 209us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.433996 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.434016 (+ 20us) server_negotiation.cc:373] Connection header received 0927 22:54:04.434093 (+ 77us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.434099 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.434183 (+ 84us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.434277 (+ 94us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:54:04.434766 (+ 489us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.435004 (+ 238us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.436908 (+ 1904us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.438191 (+ 1283us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.438291 (+ 100us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.438378 (+ 87us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.441488 (+ 3110us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.441531 (+ 43us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":66} I0927 22:54:04.442298 18935 leader_election.cc:231] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:54:04.442500 18594 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ef02af55cbde4ce3a33f3f4cb329f1b5" candidate_uuid: "54434d11164442ca96a66cda770a149c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f349d4d9e84c4596252016e4bfc620" is_pre_election: true W0927 22:54:04.444130 18786 leader_election.cc:293] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 00f349d4d9e84c4596252016e4bfc620: Illegal state: must be running to vote when last-logged opid is not known I0927 22:54:04.444510 18930 raft_consensus.cc:304] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.445571 18931 raft_consensus.cc:304] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.448086 18930 raft_consensus.cc:330] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:54:04.448211 18931 raft_consensus.cc:330] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:54:04.448341 18930 raft_consensus.cc:605] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f349d4d9e84c4596252016e4bfc620, State: Initialized, Role: FOLLOWER I0927 22:54:04.448485 18902 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.442686 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.2:41325 0927 22:54:04.442847 (+ 161us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.442861 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.442960 (+ 99us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.443558 (+ 598us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.443564 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.443594 (+ 30us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:54:04.443648 (+ 54us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.443745 (+ 97us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.443756 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.444373 (+ 617us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.444378 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.445709 (+ 1331us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.445734 (+ 25us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.447809 (+ 2075us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.447815 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.447953 (+ 138us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.448006 (+ 53us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.448187 (+ 181us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.448228 (+ 41us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":42} I0927 22:54:04.448757 18926 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.443015 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.3:59819 0927 22:54:04.443282 (+ 267us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.443289 (+ 7us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.443311 (+ 22us) server_negotiation.cc:373] Connection header received 0927 22:54:04.443368 (+ 57us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.443371 (+ 3us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.443442 (+ 71us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.443516 (+ 74us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:54:04.443903 (+ 387us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.444235 (+ 332us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.446101 (+ 1866us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.447456 (+ 1355us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.447563 (+ 107us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.447671 (+ 108us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.448368 (+ 697us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.448418 (+ 50us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":85} I0927 22:54:04.448921 18931 raft_consensus.cc:605] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b5292fbabd4fbe9b8146fa205ecb22, State: Initialized, Role: FOLLOWER I0927 22:54:04.449326 18931 consensus_queue.cc:226] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.449445 18725 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ef02af55cbde4ce3a33f3f4cb329f1b5" candidate_uuid: "54434d11164442ca96a66cda770a149c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" is_pre_election: true I0927 22:54:04.449851 18725 raft_consensus.cc:2700] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: Snoozing failure detection for 0.104s (vote granted) I0927 22:54:04.451144 18725 raft_consensus.cc:2215] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54434d11164442ca96a66cda770a149c in term 0. I0927 22:54:04.451164 18930 consensus_queue.cc:226] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.451913 18788 leader_election.cc:363] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:54:04.451982 18788 leader_election.cc:258] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0927 22:54:04.453196 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.445841 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.447994 (+ 2153us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.448011 (+ 17us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.448132 (+ 121us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.448745 (+ 613us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.448751 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.448765 (+ 14us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.448842 (+ 77us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.448907 (+ 65us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.448918 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.449576 (+ 658us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.449584 (+ 8us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.450284 (+ 700us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.450297 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.451477 (+ 1180us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.451484 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.451630 (+ 146us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.451695 (+ 65us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.452680 (+ 985us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.452691 (+ 11us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.452840 (+ 149us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.452885 (+ 45us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1986} I0927 22:54:04.453997 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.446308 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40452 0927 22:54:04.448349 (+ 2041us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.448354 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.448373 (+ 19us) server_negotiation.cc:373] Connection header received 0927 22:54:04.448424 (+ 51us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.448429 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.448605 (+ 176us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.449012 (+ 407us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.449177 (+ 165us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.449381 (+ 204us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.450464 (+ 1083us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.451310 (+ 846us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.451942 (+ 632us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.452175 (+ 233us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.452322 (+ 147us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.452616 (+ 294us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.453682 (+ 1066us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.453727 (+ 45us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1909} W0927 22:54:04.456202 18901 tablet.cc:1679] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0927 22:54:04.457810 18936 raft_consensus.cc:2700] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Snoozing failure detection for 0.101s (election complete) I0927 22:54:04.457917 18936 raft_consensus.cc:2507] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0927 22:54:04.458009 18936 raft_consensus.cc:436] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0927 22:54:04.458086 18936 raft_consensus.cc:2700] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Snoozing failure detection for 0.103s (starting election) I0927 22:54:04.458156 18936 raft_consensus.cc:2752] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Advancing to term 1 I0927 22:54:04.460033 18931 ts_tablet_manager.cc:983] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: Time spent starting tablet: real 0.047s user 0.009s sys 0.003s I0927 22:54:04.460664 18936 raft_consensus.cc:458] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.461344 18936 leader_election.cc:231] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 election: Requesting vote from peer 00f349d4d9e84c4596252016e4bfc620 I0927 22:54:04.461498 18936 leader_election.cc:231] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 election: Requesting vote from peer 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:54:04.462851 18594 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ef02af55cbde4ce3a33f3f4cb329f1b5" candidate_uuid: "54434d11164442ca96a66cda770a149c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f349d4d9e84c4596252016e4bfc620" I0927 22:54:04.463492 18725 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ef02af55cbde4ce3a33f3f4cb329f1b5" candidate_uuid: "54434d11164442ca96a66cda770a149c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" I0927 22:54:04.463656 18725 raft_consensus.cc:2752] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Advancing to term 1 I0927 22:54:04.463783 18725 raft_consensus.cc:2700] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: Snoozing failure detection for 0.103s (vote granted) I0927 22:54:04.465991 18930 ts_tablet_manager.cc:983] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Time spent starting tablet: real 0.047s user 0.003s sys 0.006s I0927 22:54:04.465987 18725 raft_consensus.cc:2215] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54434d11164442ca96a66cda770a149c in term 1. I0927 22:54:04.461794 18938 raft_consensus.cc:436] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0927 22:54:04.466269 18938 raft_consensus.cc:2700] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Snoozing failure detection for 0.110s (starting election) I0927 22:54:04.466393 18938 raft_consensus.cc:458] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.466593 18788 leader_election.cc:363] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 election: Vote granted by peer 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:54:04.466646 18788 leader_election.cc:258] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0927 22:54:04.466871 18936 raft_consensus.cc:2700] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Snoozing failure detection for 0.087s (election complete) I0927 22:54:04.466949 18936 raft_consensus.cc:2507] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Leader election won for term 1 I0927 22:54:04.467097 18936 raft_consensus.cc:572] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 1 LEADER]: Becoming Leader. State: Replica: 54434d11164442ca96a66cda770a149c, State: Running, Role: LEADER I0927 22:54:04.467495 18594 raft_consensus.cc:2752] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Advancing to term 1 I0927 22:54:04.467607 18594 raft_consensus.cc:2700] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Snoozing failure detection for 0.088s (vote granted) I0927 22:54:04.467504 18936 consensus_queue.cc:203] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.468302 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.462880 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.463497 (+ 617us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.463510 (+ 13us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.463622 (+ 112us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.464030 (+ 408us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.464035 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.464048 (+ 13us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.464131 (+ 83us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.464219 (+ 88us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.464232 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.464902 (+ 670us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.464907 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.465555 (+ 648us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.465568 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.466777 (+ 1209us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.466784 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.466907 (+ 123us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.466966 (+ 59us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.467787 (+ 821us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.467795 (+ 8us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.467921 (+ 126us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.467964 (+ 43us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":455} I0927 22:54:04.468441 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.463191 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40454 0927 22:54:04.463404 (+ 213us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.463409 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.463723 (+ 314us) server_negotiation.cc:373] Connection header received 0927 22:54:04.463803 (+ 80us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.463808 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.463897 (+ 89us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.464335 (+ 438us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.464515 (+ 180us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.464742 (+ 227us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.465741 (+ 999us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.466614 (+ 873us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.466709 (+ 95us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.467326 (+ 617us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.467473 (+ 147us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.467632 (+ 159us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.468032 (+ 400us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.468090 (+ 58us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":68} I0927 22:54:04.470160 18594 raft_consensus.cc:2215] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54434d11164442ca96a66cda770a149c in term 1. I0927 22:54:04.470588 18938 leader_election.cc:231] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:54:04.471010 18786 leader_election.cc:363] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 election: Vote granted by peer 00f349d4d9e84c4596252016e4bfc620 I0927 22:54:04.471688 18938 leader_election.cc:231] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 54434d11164442ca96a66cda770a149c I0927 22:54:04.473609 18322 catalog_manager.cc:3595] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c reported cstate change: term changed from 0 to 1, leader changed from to 54434d11164442ca96a66cda770a149c (127.17.51.3). New cstate: current_term: 1 leader_uuid: "54434d11164442ca96a66cda770a149c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } health_report { overall_health: HEALTHY } } } I0927 22:54:04.479010 18926 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.471172 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.1:48867 0927 22:54:04.471367 (+ 195us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.471373 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.472811 (+ 1438us) server_negotiation.cc:373] Connection header received 0927 22:54:04.473094 (+ 283us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.473099 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.473195 (+ 96us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.473267 (+ 72us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:54:04.473991 (+ 724us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.474258 (+ 267us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.476462 (+ 2204us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.477729 (+ 1267us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.477829 (+ 100us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.477926 (+ 97us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.478671 (+ 745us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.478718 (+ 47us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":58} I0927 22:54:04.479621 18940 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.471118 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.2:41325 0927 22:54:04.472730 (+ 1612us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.472758 (+ 28us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.472917 (+ 159us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.473453 (+ 536us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.473459 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.473494 (+ 35us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:54:04.473622 (+ 128us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.473835 (+ 213us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.473852 (+ 17us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.474416 (+ 564us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.474421 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.475803 (+ 1382us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.475816 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.478097 (+ 2281us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.478104 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.478260 (+ 156us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.478315 (+ 55us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.479213 (+ 898us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.479260 (+ 47us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":334,"threads_started":1} I0927 22:54:04.480368 18725 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ef02af55cbde4ce3a33f3f4cb329f1b5" candidate_uuid: "00f349d4d9e84c4596252016e4bfc620" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" is_pre_election: true I0927 22:54:04.480628 18725 raft_consensus.cc:2145] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00f349d4d9e84c4596252016e4bfc620 in current term 1: Already voted for candidate 54434d11164442ca96a66cda770a149c in this term. I0927 22:54:04.481418 18528 leader_election.cc:377] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 26b5292fbabd4fbe9b8146fa205ecb22. Message: Invalid argument: T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 00f349d4d9e84c4596252016e4bfc620 in current term 1: Already voted for candidate 54434d11164442ca96a66cda770a149c in this term. I0927 22:54:04.485673 18925 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.475135 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.1:50533 0927 22:54:04.475325 (+ 190us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.475331 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.476289 (+ 958us) server_negotiation.cc:373] Connection header received 0927 22:54:04.476435 (+ 146us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.476441 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.476558 (+ 117us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.476640 (+ 82us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:54:04.479676 (+ 3036us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.479912 (+ 236us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.481669 (+ 1757us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.484142 (+ 2473us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.484229 (+ 87us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.484341 (+ 112us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.485371 (+ 1030us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.485420 (+ 49us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":39} I0927 22:54:04.486340 18941 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.473922 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.3:44871 0927 22:54:04.476205 (+ 2283us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.476216 (+ 11us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.476312 (+ 96us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.476776 (+ 464us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.476781 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.476810 (+ 29us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:54:04.476920 (+ 110us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.479532 (+ 2612us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.479545 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.480076 (+ 531us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.480081 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.481516 (+ 1435us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.481529 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.484733 (+ 3204us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.484739 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.484925 (+ 186us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.484975 (+ 50us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.485814 (+ 839us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.485856 (+ 42us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":2164,"mutex_wait_us":1,"thread_start_us":2039,"threads_started":1} I0927 22:54:04.487450 18854 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "ef02af55cbde4ce3a33f3f4cb329f1b5" candidate_uuid: "00f349d4d9e84c4596252016e4bfc620" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54434d11164442ca96a66cda770a149c" is_pre_election: true I0927 22:54:04.487768 18854 raft_consensus.cc:2176] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 00f349d4d9e84c4596252016e4bfc620 for term 1 because replica is either leader or believes a valid leader to be alive. I0927 22:54:04.488205 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.477540 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.478430 (+ 890us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.478445 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.478556 (+ 111us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.479033 (+ 477us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.479038 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.479051 (+ 13us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.479133 (+ 82us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.479199 (+ 66us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.479210 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.480138 (+ 928us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.480144 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.481048 (+ 904us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.481079 (+ 31us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.486472 (+ 5393us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.486479 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.486607 (+ 128us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.486676 (+ 69us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.487632 (+ 956us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.487644 (+ 12us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.487797 (+ 153us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.487885 (+ 88us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":720} I0927 22:54:04.488332 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.478027 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40456 0927 22:54:04.478226 (+ 199us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.478232 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.478653 (+ 421us) server_negotiation.cc:373] Connection header received 0927 22:54:04.478717 (+ 64us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.478723 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.478813 (+ 90us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.478893 (+ 80us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.479369 (+ 476us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.479953 (+ 584us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.481268 (+ 1315us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.486184 (+ 4916us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.486297 (+ 113us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.487195 (+ 898us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.487382 (+ 187us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.487545 (+ 163us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.487893 (+ 348us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.487934 (+ 41us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":80} I0927 22:54:04.488687 18529 leader_election.cc:377] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 54434d11164442ca96a66cda770a149c. Message: Invalid argument: T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 00f349d4d9e84c4596252016e4bfc620 for term 1 because replica is either leader or believes a valid leader to be alive. I0927 22:54:04.488797 18529 leader_election.cc:258] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0927 22:54:04.489127 18938 raft_consensus.cc:2700] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Snoozing failure detection for 0.087s (election complete) I0927 22:54:04.489204 18938 raft_consensus.cc:2455] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0927 22:54:04.493958 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.490033 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.490309 (+ 276us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.490324 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.490416 (+ 92us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.490778 (+ 362us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.490781 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.490791 (+ 10us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.490838 (+ 47us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.490881 (+ 43us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.490888 (+ 7us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.491294 (+ 406us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.491299 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.491899 (+ 600us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.491912 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.492759 (+ 847us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.492766 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.492894 (+ 128us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.492951 (+ 57us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.493582 (+ 631us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.493590 (+ 8us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.493721 (+ 131us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.493753 (+ 32us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":72} I0927 22:54:04.493991 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.490142 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40458 0927 22:54:04.490331 (+ 189us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.490336 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.490356 (+ 20us) server_negotiation.cc:373] Connection header received 0927 22:54:04.490551 (+ 195us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.490555 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.490647 (+ 92us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.490769 (+ 122us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.491000 (+ 231us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.491154 (+ 154us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.492036 (+ 882us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.492606 (+ 570us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.492689 (+ 83us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.493241 (+ 552us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.493362 (+ 121us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.493495 (+ 133us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.493784 (+ 289us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.493812 (+ 28us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":73} I0927 22:54:04.497155 18594 raft_consensus.cc:1058] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Refusing update from remote peer 54434d11164442ca96a66cda770a149c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0927 22:54:04.498132 18936 consensus_queue.cc:962] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [LEADER]: Connected to new peer: Peer: permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0927 22:54:04.501420 18725 raft_consensus.cc:1058] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Refusing update from remote peer 54434d11164442ca96a66cda770a149c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0927 22:54:04.502224 18936 consensus_queue.cc:962] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0927 22:54:04.503753 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.497907 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.498117 (+ 210us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.498128 (+ 11us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.498201 (+ 73us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.498508 (+ 307us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.498513 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.498533 (+ 20us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.498593 (+ 60us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.498658 (+ 65us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.498669 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.500228 (+ 1559us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.500234 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.500965 (+ 731us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.500977 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.502237 (+ 1260us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.502246 (+ 9us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.502378 (+ 132us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.502438 (+ 60us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.503281 (+ 843us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.503293 (+ 12us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.503440 (+ 147us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.503485 (+ 45us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":91} I0927 22:54:04.515619 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.497966 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40460 0927 22:54:04.498157 (+ 191us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.498162 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.498182 (+ 20us) server_negotiation.cc:373] Connection header received 0927 22:54:04.498305 (+ 123us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.498310 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.498391 (+ 81us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.498491 (+ 100us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.498798 (+ 307us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.499026 (+ 228us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.501144 (+ 2118us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.501945 (+ 801us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.502043 (+ 98us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.502765 (+ 722us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.502908 (+ 143us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.503086 (+ 178us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.504161 (+ 1075us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.504518 (+ 357us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":67} I0927 22:54:04.525207 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.520974 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.3:44871 0927 22:54:04.521348 (+ 374us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.521362 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.521471 (+ 109us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.521850 (+ 379us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.521854 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.521866 (+ 12us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.521947 (+ 81us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.522023 (+ 76us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.522031 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.522498 (+ 467us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.522501 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.522985 (+ 484us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.522995 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.523770 (+ 775us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.523775 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.523903 (+ 128us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.523965 (+ 62us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.524792 (+ 827us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.524800 (+ 8us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.524939 (+ 139us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.524974 (+ 35us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":238} I0927 22:54:04.525298 18925 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.520975 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42154 0927 22:54:04.521187 (+ 212us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.521193 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.521439 (+ 246us) server_negotiation.cc:373] Connection header received 0927 22:54:04.521588 (+ 149us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.521592 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.521677 (+ 85us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.521774 (+ 97us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.522154 (+ 380us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.522364 (+ 210us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.523111 (+ 747us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.523612 (+ 501us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.523693 (+ 81us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.524272 (+ 579us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.524605 (+ 333us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.524745 (+ 140us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.525006 (+ 261us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.525040 (+ 34us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":51} W0927 22:54:04.525676 18644 tablet.cc:1679] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Can't schedule compaction. Clean time has not been advanced past its initial value. I0927 22:54:04.551786 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.547445 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.3:44871 0927 22:54:04.547744 (+ 299us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.547763 (+ 19us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.547879 (+ 116us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.548338 (+ 459us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.548341 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.548352 (+ 11us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.548412 (+ 60us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.548504 (+ 92us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.548515 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.549043 (+ 528us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.549046 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.549588 (+ 542us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.549605 (+ 17us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.550653 (+ 1048us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.550659 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.550788 (+ 129us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.550855 (+ 67us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.551399 (+ 544us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.551408 (+ 9us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.551509 (+ 101us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.551544 (+ 35us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":93} I0927 22:54:04.551846 18925 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.547547 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42156 0927 22:54:04.547757 (+ 210us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.547763 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.547807 (+ 44us) server_negotiation.cc:373] Connection header received 0927 22:54:04.548025 (+ 218us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.548031 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.548135 (+ 104us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.548194 (+ 59us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.548652 (+ 458us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.548895 (+ 243us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.549724 (+ 829us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.550469 (+ 745us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.550591 (+ 122us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.551100 (+ 509us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.551219 (+ 119us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.551349 (+ 130us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.551581 (+ 232us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.551620 (+ 39us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":62} I0927 22:54:04.565871 18913 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.560928 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.561268 (+ 340us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.561286 (+ 18us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.561414 (+ 128us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.561735 (+ 321us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.561741 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.561756 (+ 15us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.561831 (+ 75us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.561916 (+ 85us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.561928 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.562373 (+ 445us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.562379 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.563127 (+ 748us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.563140 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.564248 (+ 1108us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.564255 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.564391 (+ 136us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.564456 (+ 65us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.565309 (+ 853us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.565321 (+ 12us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.565481 (+ 160us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.565528 (+ 47us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":177} I0927 22:54:04.565943 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.560970 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40466 0927 22:54:04.561163 (+ 193us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.561167 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.561341 (+ 174us) server_negotiation.cc:373] Connection header received 0927 22:54:04.561522 (+ 181us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.561526 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.561623 (+ 97us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.561698 (+ 75us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.562030 (+ 332us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.562257 (+ 227us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.563275 (+ 1018us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.564039 (+ 764us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.564181 (+ 142us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.564862 (+ 681us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.565049 (+ 187us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.565243 (+ 194us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.565584 (+ 341us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.565646 (+ 62us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":70} I0927 22:54:04.566860 18322 catalog_manager.cc:1666] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40466: table { table_name: "test-table" } I0927 22:54:04.577226 18322 catalog_manager.cc:3806] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Sending DeleteTablet for 3 replicas of tablet ef02af55cbde4ce3a33f3f4cb329f1b5 I0927 22:54:04.577422 18322 catalog_manager.cc:2922] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ef02af55cbde4ce3a33f3f4cb329f1b5 on 00f349d4d9e84c4596252016e4bfc620 (127.17.51.1:40273) (Table deleted at 2018-09-27 22:54:04 UTC) I0927 22:54:04.577702 18322 catalog_manager.cc:2922] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ef02af55cbde4ce3a33f3f4cb329f1b5 on 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2:41325) (Table deleted at 2018-09-27 22:54:04 UTC) I0927 22:54:04.577875 18322 catalog_manager.cc:2922] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet ef02af55cbde4ce3a33f3f4cb329f1b5 on 54434d11164442ca96a66cda770a149c (127.17.51.3:44871) (Table deleted at 2018-09-27 22:54:04 UTC) I0927 22:54:04.579277 18834 tablet_service.cc:799] Processing DeleteTablet for tablet ef02af55cbde4ce3a33f3f4cb329f1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-09-27 22:54:04 UTC) from {username='kudu', principal='kudu/127.17.51.62@KRBTEST.COM'} at 127.0.0.1:42128 I0927 22:54:04.579860 18834 tablet_replica.cc:262] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: stopping tablet replica I0927 22:54:04.580103 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op CompactRowSetsOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.580070 18705 tablet_service.cc:799] Processing DeleteTablet for tablet ef02af55cbde4ce3a33f3f4cb329f1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-09-27 22:54:04 UTC) from {username='kudu', principal='kudu/127.17.51.62@KRBTEST.COM'} at 127.0.0.1:40738 I0927 22:54:04.580226 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op MinorDeltaCompactionOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.580304 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op MajorDeltaCompactionOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.580317 18705 tablet_replica.cc:262] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: stopping tablet replica I0927 22:54:04.580404 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op UndoDeltaBlockGCOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.580507 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op CompactRowSetsOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.580521 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op FlushMRSOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.580619 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op FlushDeltaMemStoresOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.580624 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op MinorDeltaCompactionOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.580679 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op LogGCOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.580716 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op MajorDeltaCompactionOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.580811 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op UndoDeltaBlockGCOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.580799 18834 raft_consensus.cc:2012] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 1 LEADER]: Raft consensus shutting down. I0927 22:54:04.580893 18834 consensus_peers.cc:457] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c -> Peer 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2:41325): Closing peer: 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:54:04.580946 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op FlushMRSOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.581018 18834 consensus_peers.cc:457] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c -> Peer 00f349d4d9e84c4596252016e4bfc620 (127.17.51.1:40273): Closing peer: 00f349d4d9e84c4596252016e4bfc620 I0927 22:54:04.581048 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op FlushDeltaMemStoresOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.581030 18574 tablet_service.cc:799] Processing DeleteTablet for tablet ef02af55cbde4ce3a33f3f4cb329f1b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-09-27 22:54:04 UTC) from {username='kudu', principal='kudu/127.17.51.62@KRBTEST.COM'} at 127.0.0.1:32812 I0927 22:54:04.581145 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op LogGCOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.581256 18705 raft_consensus.cc:2012] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Raft consensus shutting down. I0927 22:54:04.581295 18574 tablet_replica.cc:262] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: stopping tablet replica I0927 22:54:04.581439 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op CompactRowSetsOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.581465 18834 raft_consensus.cc:2039] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Raft consensus is shut down! I0927 22:54:04.581547 18705 raft_consensus.cc:2039] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Raft consensus is shut down! I0927 22:54:04.581758 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op MinorDeltaCompactionOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.581856 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op MajorDeltaCompactionOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.582005 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op UndoDeltaBlockGCOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.582123 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op FlushMRSOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.582207 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op FlushDeltaMemStoresOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.582267 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op LogGCOp(ef02af55cbde4ce3a33f3f4cb329f1b5) I0927 22:54:04.582368 18574 raft_consensus.cc:2012] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Raft consensus shutting down. I0927 22:54:04.582609 18574 raft_consensus.cc:2039] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Raft consensus is shut down! I0927 22:54:04.584280 18705 ts_tablet_manager.cc:1277] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: Deleting tablet data with delete state TABLET_DATA_DELETED I0927 22:54:04.588064 18705 ts_tablet_manager.cc:1290] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0927 22:54:04.588166 18705 log.cc:981] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: Deleting WAL directory at /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/wal/wals/ef02af55cbde4ce3a33f3f4cb329f1b5 I0927 22:54:04.589162 18705 ts_tablet_manager.cc:1310] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 26b5292fbabd4fbe9b8146fa205ecb22: Deleting consensus metadata I0927 22:54:04.590256 18834 ts_tablet_manager.cc:1277] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Deleting tablet data with delete state TABLET_DATA_DELETED I0927 22:54:04.591037 18308 catalog_manager.cc:2900] TS 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2:41325): tablet ef02af55cbde4ce3a33f3f4cb329f1b5 (table test-table [id=40586b0786544a20b9f645e03b20463b]) successfully deleted I0927 22:54:04.592123 18574 ts_tablet_manager.cc:1277] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Deleting tablet data with delete state TABLET_DATA_DELETED I0927 22:54:04.595582 18834 ts_tablet_manager.cc:1290] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0927 22:54:04.595823 18834 log.cc:981] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Deleting WAL directory at /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/wal/wals/ef02af55cbde4ce3a33f3f4cb329f1b5 I0927 22:54:04.596374 18834 ts_tablet_manager.cc:1310] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 54434d11164442ca96a66cda770a149c: Deleting consensus metadata I0927 22:54:04.598564 18574 ts_tablet_manager.cc:1290] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0927 22:54:04.598743 18574 log.cc:981] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Deleting WAL directory at /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/wal/wals/ef02af55cbde4ce3a33f3f4cb329f1b5 I0927 22:54:04.598764 18309 catalog_manager.cc:2900] TS 54434d11164442ca96a66cda770a149c (127.17.51.3:44871): tablet ef02af55cbde4ce3a33f3f4cb329f1b5 (table test-table [id=40586b0786544a20b9f645e03b20463b]) successfully deleted I0927 22:54:04.601833 18574 ts_tablet_manager.cc:1310] T ef02af55cbde4ce3a33f3f4cb329f1b5 P 00f349d4d9e84c4596252016e4bfc620: Deleting consensus metadata I0927 22:54:04.603404 18306 catalog_manager.cc:2900] TS 00f349d4d9e84c4596252016e4bfc620 (127.17.51.1:40273): tablet ef02af55cbde4ce3a33f3f4cb329f1b5 (table test-table [id=40586b0786544a20b9f645e03b20463b]) successfully deleted I0927 22:54:04.630000 18961 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.607154 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:54:04.611233 (+ 4079us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.611248 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.611431 (+ 183us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.611940 (+ 509us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.611945 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.613573 (+ 1628us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.613707 (+ 134us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.613943 (+ 236us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.613956 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.614616 (+ 660us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.614621 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.615563 (+ 942us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.615576 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.617235 (+ 1659us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.617242 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.617392 (+ 150us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.617683 (+ 291us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.617686 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.619019 (+ 1333us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.624543 (+ 5524us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.624549 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.624553 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.625318 (+ 765us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.627424 (+ 2106us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.627429 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.627433 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.627507 (+ 74us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.628872 (+ 1365us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.628878 (+ 6us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.629070 (+ 192us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.629309 (+ 239us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.629497 (+ 188us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":3824,"thread_start_us":3660,"threads_started":1} I0927 22:54:04.631719 18905 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.606770 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42562 0927 22:54:04.606973 (+ 203us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.606979 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.611309 (+ 4330us) server_negotiation.cc:373] Connection header received 0927 22:54:04.611588 (+ 279us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.611594 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.611682 (+ 88us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.611760 (+ 78us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.614124 (+ 2364us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.614430 (+ 306us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.615765 (+ 1335us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.616854 (+ 1089us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.616970 (+ 116us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.620897 (+ 3927us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.620930 (+ 33us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.620934 (+ 4us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.620974 (+ 40us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.624224 (+ 3250us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.625620 (+ 1396us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.625626 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.625629 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.625972 (+ 343us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.627832 (+ 1860us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.627838 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.627841 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.628200 (+ 359us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.628601 (+ 401us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.630565 (+ 1964us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.631334 (+ 769us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":66} I0927 22:54:04.637706 18960 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.605502 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.606477 (+ 975us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.606499 (+ 22us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.606678 (+ 179us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.609150 (+ 2472us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.609155 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.611049 (+ 1894us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.611178 (+ 129us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.611339 (+ 161us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.611351 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.611926 (+ 575us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.611931 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.612692 (+ 761us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.612705 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.614156 (+ 1451us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.614163 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.614288 (+ 125us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.614473 (+ 185us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.614477 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.616230 (+ 1753us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.634121 (+ 17891us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.634126 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.634130 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.634542 (+ 412us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.635626 (+ 1084us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.635631 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.635634 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.635707 (+ 73us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.636685 (+ 978us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.636690 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.636834 (+ 144us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.637043 (+ 209us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.637225 (+ 182us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":3189,"threads_started":1} I0927 22:54:04.638386 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.605534 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40468 0927 22:54:04.605736 (+ 202us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.605742 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.606557 (+ 815us) server_negotiation.cc:373] Connection header received 0927 22:54:04.606831 (+ 274us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.606837 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.606936 (+ 99us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.607041 (+ 105us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.611502 (+ 4461us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.611762 (+ 260us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.612853 (+ 1091us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.613691 (+ 838us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.613780 (+ 89us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.616863 (+ 3083us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.616894 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.616900 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.616939 (+ 39us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.633814 (+ 16875us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.634872 (+ 1058us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.634878 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.634881 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.635288 (+ 407us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.635975 (+ 687us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.635981 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.635984 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.636307 (+ 323us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.636749 (+ 442us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.637240 (+ 491us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.638018 (+ 778us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":64} I0927 22:54:04.638483 18959 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.605976 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:54:04.609026 (+ 3050us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.609040 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.609210 (+ 170us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.609746 (+ 536us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.609751 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.611895 (+ 2144us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:54:04.612015 (+ 120us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.612298 (+ 283us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.612310 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.613039 (+ 729us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.613043 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.613890 (+ 847us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.613906 (+ 16us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.615518 (+ 1612us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.615526 (+ 8us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.615704 (+ 178us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.615960 (+ 256us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:54:04.615964 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:54:04.619053 (+ 3089us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:54:04.634314 (+ 15261us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.634319 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.634322 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.634922 (+ 600us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.635988 (+ 1066us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:54:04.635992 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:54:04.635996 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:54:04.636072 (+ 76us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:54:04.637333 (+ 1261us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:54:04.637342 (+ 9us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:54:04.637480 (+ 138us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.637725 (+ 245us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.638067 (+ 342us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":2871,"mutex_wait_us":1712,"thread_start_us":212,"threads_started":1} I0927 22:54:04.639938 18906 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.605629 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51264 0927 22:54:04.605853 (+ 224us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.605859 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.609107 (+ 3248us) server_negotiation.cc:373] Connection header received 0927 22:54:04.609354 (+ 247us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.609361 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.609452 (+ 91us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.609533 (+ 81us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:54:04.612458 (+ 2925us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.612837 (+ 379us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.614146 (+ 1309us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.615089 (+ 943us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.615179 (+ 90us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.619545 (+ 4366us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:54:04.619576 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:54:04.619594 (+ 18us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:54:04.619633 (+ 39us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:54:04.632195 (+ 12562us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.635274 (+ 3079us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.635280 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.635283 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.635691 (+ 408us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:54:04.636330 (+ 639us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:54:04.636335 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:54:04.636338 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:54:04.636684 (+ 346us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:54:04.637042 (+ 358us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.638805 (+ 1763us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.639533 (+ 728us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":70} I0927 22:54:04.646836 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.641927 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40474 0927 22:54:04.642153 (+ 226us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.642158 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.642178 (+ 20us) server_negotiation.cc:373] Connection header received 0927 22:54:04.642233 (+ 55us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.642238 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.642329 (+ 91us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.642416 (+ 87us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.642940 (+ 524us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.643889 (+ 949us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.644988 (+ 1099us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.645486 (+ 498us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.645570 (+ 84us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.646091 (+ 521us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.646202 (+ 111us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.646313 (+ 111us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.646630 (+ 317us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.646660 (+ 30us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":64} I0927 22:54:04.646924 18959 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.641709 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.641923 (+ 214us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.641941 (+ 18us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.642043 (+ 102us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.642645 (+ 602us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.642651 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.642672 (+ 21us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.642743 (+ 71us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.642812 (+ 69us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.642822 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.644012 (+ 1190us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.644017 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.644862 (+ 845us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.644875 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.645628 (+ 753us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.645634 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.645763 (+ 129us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.645827 (+ 64us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.646411 (+ 584us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.646422 (+ 11us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.646586 (+ 164us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.646627 (+ 41us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":41} I0927 22:54:04.647666 18322 catalog_manager.cc:1294] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40474: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "val" type: INT32 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0927 22:54:04.648289 18322 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0927 22:54:04.665469 18959 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.659439 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.660287 (+ 848us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.660304 (+ 17us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.660404 (+ 100us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.661014 (+ 610us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.661019 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.661032 (+ 13us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.661134 (+ 102us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.661219 (+ 85us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.661232 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.661836 (+ 604us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.661841 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.662554 (+ 713us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.662567 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.663802 (+ 1235us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.663809 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.663947 (+ 138us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.664016 (+ 69us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.664880 (+ 864us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.664891 (+ 11us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.665036 (+ 145us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.665105 (+ 69us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":683} I0927 22:54:04.665390 18834 tablet_service.cc:758] Processing CreateTablet for tablet 224a68a39c264e3d85e18d8f840e57ce (table=test-table [id=53149fe3c1184e0e87ab180fa11b73de]), partition=RANGE (key) PARTITION UNBOUNDED I0927 22:54:04.665890 18834 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 224a68a39c264e3d85e18d8f840e57ce. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:54:04.666163 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.659654 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40476 0927 22:54:04.660613 (+ 959us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.660618 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.660638 (+ 20us) server_negotiation.cc:373] Connection header received 0927 22:54:04.660689 (+ 51us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.660695 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.660779 (+ 84us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.660854 (+ 75us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.661376 (+ 522us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.661669 (+ 293us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.662721 (+ 1052us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.663551 (+ 830us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.663650 (+ 99us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.664394 (+ 744us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.664566 (+ 172us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.664720 (+ 154us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.665834 (+ 1114us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.665876 (+ 42us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":838} I0927 22:54:04.667505 18705 tablet_service.cc:758] Processing CreateTablet for tablet 224a68a39c264e3d85e18d8f840e57ce (table=test-table [id=53149fe3c1184e0e87ab180fa11b73de]), partition=RANGE (key) PARTITION UNBOUNDED I0927 22:54:04.667860 18705 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 224a68a39c264e3d85e18d8f840e57ce. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:54:04.669893 18834 ts_tablet_manager.cc:1080] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: Registered tablet (data state: TABLET_DATA_READY) I0927 22:54:04.670181 18929 ts_tablet_manager.cc:938] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: Bootstrapping tablet I0927 22:54:04.671509 18929 tablet_bootstrap.cc:436] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: Bootstrap starting. I0927 22:54:04.672420 18705 ts_tablet_manager.cc:1080] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: Registered tablet (data state: TABLET_DATA_READY) I0927 22:54:04.673038 18929 tablet_bootstrap.cc:581] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: No blocks or log segments found. Creating new log. I0927 22:54:04.674158 18574 tablet_service.cc:758] Processing CreateTablet for tablet 224a68a39c264e3d85e18d8f840e57ce (table=test-table [id=53149fe3c1184e0e87ab180fa11b73de]), partition=RANGE (key) PARTITION UNBOUNDED I0927 22:54:04.674501 18574 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet 224a68a39c264e3d85e18d8f840e57ce. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:54:04.677446 18929 tablet_bootstrap.cc:436] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: No bootstrap required, opened a new log I0927 22:54:04.677610 18929 ts_tablet_manager.cc:955] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s I0927 22:54:04.677973 18574 ts_tablet_manager.cc:1080] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: Registered tablet (data state: TABLET_DATA_READY) I0927 22:54:04.678203 18930 ts_tablet_manager.cc:938] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: Bootstrapping tablet I0927 22:54:04.678355 18930 tablet_bootstrap.cc:436] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: Bootstrap starting. I0927 22:54:04.678588 18929 raft_consensus.cc:304] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.678875 18929 raft_consensus.cc:330] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:54:04.678962 18929 raft_consensus.cc:605] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54434d11164442ca96a66cda770a149c, State: Initialized, Role: FOLLOWER I0927 22:54:04.679214 18929 consensus_queue.cc:226] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.679715 18930 tablet_bootstrap.cc:581] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: No blocks or log segments found. Creating new log. I0927 22:54:04.686277 18929 ts_tablet_manager.cc:983] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s I0927 22:54:04.688661 18931 ts_tablet_manager.cc:938] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: Bootstrapping tablet I0927 22:54:04.688846 18931 tablet_bootstrap.cc:436] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: Bootstrap starting. I0927 22:54:04.689999 18930 tablet_bootstrap.cc:436] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: No bootstrap required, opened a new log I0927 22:54:04.690124 18930 ts_tablet_manager.cc:955] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s I0927 22:54:04.690440 18931 tablet_bootstrap.cc:581] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: No blocks or log segments found. Creating new log. I0927 22:54:04.690732 18930 raft_consensus.cc:304] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.690881 18930 raft_consensus.cc:330] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:54:04.690917 18930 raft_consensus.cc:605] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f349d4d9e84c4596252016e4bfc620, State: Initialized, Role: FOLLOWER I0927 22:54:04.691071 18930 consensus_queue.cc:226] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.692909 18930 ts_tablet_manager.cc:983] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s I0927 22:54:04.694530 18945 raft_consensus.cc:436] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0927 22:54:04.694622 18945 raft_consensus.cc:2700] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: Snoozing failure detection for 0.102s (starting election) I0927 22:54:04.694730 18945 raft_consensus.cc:458] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.695420 18945 leader_election.cc:231] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:54:04.695612 18945 leader_election.cc:231] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 54434d11164442ca96a66cda770a149c I0927 22:54:04.696543 18854 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "224a68a39c264e3d85e18d8f840e57ce" candidate_uuid: "00f349d4d9e84c4596252016e4bfc620" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54434d11164442ca96a66cda770a149c" is_pre_election: true I0927 22:54:04.696739 18854 raft_consensus.cc:2700] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: Snoozing failure detection for 0.091s (vote granted) I0927 22:54:04.696842 18854 raft_consensus.cc:2215] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00f349d4d9e84c4596252016e4bfc620 in term 0. I0927 22:54:04.697068 18931 tablet_bootstrap.cc:436] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: No bootstrap required, opened a new log I0927 22:54:04.697208 18931 ts_tablet_manager.cc:955] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s I0927 22:54:04.697580 18529 leader_election.cc:363] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 54434d11164442ca96a66cda770a149c I0927 22:54:04.697646 18529 leader_election.cc:258] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0927 22:54:04.697888 18945 raft_consensus.cc:2700] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: Snoozing failure detection for 0.101s (election complete) I0927 22:54:04.697969 18945 raft_consensus.cc:2507] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0927 22:54:04.698014 18945 raft_consensus.cc:436] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0927 22:54:04.697913 18931 raft_consensus.cc:304] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.698097 18945 raft_consensus.cc:2700] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: Snoozing failure detection for 0.080s (starting election) I0927 22:54:04.698144 18931 raft_consensus.cc:330] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:54:04.698150 18945 raft_consensus.cc:2752] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Advancing to term 1 I0927 22:54:04.698194 18931 raft_consensus.cc:605] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b5292fbabd4fbe9b8146fa205ecb22, State: Initialized, Role: FOLLOWER I0927 22:54:04.698501 18959 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.669298 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.669493 (+ 195us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.669507 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.669590 (+ 83us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.693820 (+ 24230us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.693828 (+ 8us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.693849 (+ 21us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.693971 (+ 122us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.694053 (+ 82us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.694084 (+ 31us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.694686 (+ 602us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.694691 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.695424 (+ 733us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.695437 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.696760 (+ 1323us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.696767 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.697000 (+ 233us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.697099 (+ 99us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.697948 (+ 849us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.697961 (+ 13us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.698125 (+ 164us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.698180 (+ 55us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":64} I0927 22:54:04.698405 18931 consensus_queue.cc:226] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.700767 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.672912 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40478 0927 22:54:04.693265 (+ 20353us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.693272 (+ 7us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.693295 (+ 23us) server_negotiation.cc:373] Connection header received 0927 22:54:04.693382 (+ 87us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.693387 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.693486 (+ 99us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.693605 (+ 119us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.694262 (+ 657us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.694516 (+ 254us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.695601 (+ 1085us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.696407 (+ 806us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.696594 (+ 187us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.697448 (+ 854us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.697620 (+ 172us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.697780 (+ 160us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.698922 (+ 1142us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.698967 (+ 45us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":20211} I0927 22:54:04.702795 18725 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "224a68a39c264e3d85e18d8f840e57ce" candidate_uuid: "00f349d4d9e84c4596252016e4bfc620" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" is_pre_election: true I0927 22:54:04.702970 18725 raft_consensus.cc:2700] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: Snoozing failure detection for 0.076s (vote granted) I0927 22:54:04.703110 18725 raft_consensus.cc:2215] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00f349d4d9e84c4596252016e4bfc620 in term 0. I0927 22:54:04.704639 18528 leader_election.cc:363] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:54:04.705549 18945 raft_consensus.cc:458] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.706151 18945 leader_election.cc:231] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 election: Requesting vote from peer 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:54:04.707413 18945 leader_election.cc:231] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 election: Requesting vote from peer 54434d11164442ca96a66cda770a149c I0927 22:54:04.707551 18931 ts_tablet_manager.cc:983] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s I0927 22:54:04.708112 18725 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "224a68a39c264e3d85e18d8f840e57ce" candidate_uuid: "00f349d4d9e84c4596252016e4bfc620" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" I0927 22:54:04.708259 18725 raft_consensus.cc:2752] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Advancing to term 1 I0927 22:54:04.708350 18725 raft_consensus.cc:2700] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: Snoozing failure detection for 0.105s (vote granted) I0927 22:54:04.709008 18854 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "224a68a39c264e3d85e18d8f840e57ce" candidate_uuid: "00f349d4d9e84c4596252016e4bfc620" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54434d11164442ca96a66cda770a149c" I0927 22:54:04.709198 18854 raft_consensus.cc:2752] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Advancing to term 1 I0927 22:54:04.709280 18854 raft_consensus.cc:2700] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: Snoozing failure detection for 0.085s (vote granted) I0927 22:54:04.710505 18725 raft_consensus.cc:2215] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f349d4d9e84c4596252016e4bfc620 in term 1. I0927 22:54:04.711169 18528 leader_election.cc:363] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 election: Vote granted by peer 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:54:04.711233 18528 leader_election.cc:258] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0927 22:54:04.711447 18945 raft_consensus.cc:2700] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: Snoozing failure detection for 0.112s (election complete) I0927 22:54:04.711514 18854 raft_consensus.cc:2215] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f349d4d9e84c4596252016e4bfc620 in term 1. I0927 22:54:04.711535 18945 raft_consensus.cc:2507] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Leader election won for term 1 I0927 22:54:04.712131 18529 leader_election.cc:363] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [CANDIDATE]: Term 1 election: Vote granted by peer 54434d11164442ca96a66cda770a149c I0927 22:54:04.712177 18945 raft_consensus.cc:572] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 1 LEADER]: Becoming Leader. State: Replica: 00f349d4d9e84c4596252016e4bfc620, State: Running, Role: LEADER I0927 22:54:04.712517 18945 consensus_queue.cc:203] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:54:04.715281 18959 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.704174 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.705252 (+ 1078us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.705270 (+ 18us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.705379 (+ 109us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.710040 (+ 4661us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.710045 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.710076 (+ 31us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.710162 (+ 86us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.710244 (+ 82us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.710256 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.710799 (+ 543us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.710804 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.712384 (+ 1580us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.712398 (+ 14us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.713737 (+ 1339us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.713744 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.713872 (+ 128us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.713939 (+ 67us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.714723 (+ 784us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.714732 (+ 9us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.714876 (+ 144us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.714925 (+ 49us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":911} I0927 22:54:04.715318 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.706297 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40480 0927 22:54:04.709652 (+ 3355us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.709657 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.709677 (+ 20us) server_negotiation.cc:373] Connection header received 0927 22:54:04.709730 (+ 53us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.709736 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.709825 (+ 89us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.709904 (+ 79us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.710401 (+ 497us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.710637 (+ 236us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.712615 (+ 1978us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.713461 (+ 846us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.713562 (+ 101us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.714306 (+ 744us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.714495 (+ 189us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.714651 (+ 156us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.714993 (+ 342us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.715037 (+ 44us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":3224} I0927 22:54:04.718482 18323 catalog_manager.cc:3595] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 reported cstate change: term changed from 0 to 1, leader changed from to 00f349d4d9e84c4596252016e4bfc620 (127.17.51.1). New cstate: current_term: 1 leader_uuid: "00f349d4d9e84c4596252016e4bfc620" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } health_report { overall_health: UNKNOWN } } } I0927 22:54:04.727869 18959 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.719278 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.721152 (+ 1874us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.721170 (+ 18us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.721278 (+ 108us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.721764 (+ 486us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.721769 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.721782 (+ 13us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.721848 (+ 66us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.721913 (+ 65us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.721924 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.722439 (+ 515us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.722443 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.723286 (+ 843us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.723300 (+ 14us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.724688 (+ 1388us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.724696 (+ 8us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.724922 (+ 226us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.724982 (+ 60us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.727195 (+ 2213us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.727203 (+ 8us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.727363 (+ 160us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.727407 (+ 44us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1701} I0927 22:54:04.727936 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.719646 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40482 0927 22:54:04.719833 (+ 187us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.719838 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.721375 (+ 1537us) server_negotiation.cc:373] Connection header received 0927 22:54:04.721440 (+ 65us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.721445 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.721529 (+ 84us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.721625 (+ 96us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.722077 (+ 452us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.722270 (+ 193us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.723450 (+ 1180us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.724408 (+ 958us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.724529 (+ 121us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.726800 (+ 2271us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.726914 (+ 114us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.727100 (+ 186us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.727512 (+ 412us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.727553 (+ 41us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":2033,"server-negotiator.queue_time_us":54} I0927 22:54:04.735560 18959 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.729784 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.729970 (+ 186us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.729987 (+ 17us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.730093 (+ 106us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.730459 (+ 366us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.730462 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.730471 (+ 9us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.730536 (+ 65us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.730620 (+ 84us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.730628 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.731202 (+ 574us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.731208 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.732046 (+ 838us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.732073 (+ 27us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.734034 (+ 1961us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.734041 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.734216 (+ 175us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.734275 (+ 59us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.734993 (+ 718us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.735001 (+ 8us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.735131 (+ 130us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.735162 (+ 31us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":55} I0927 22:54:04.735652 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.729984 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40484 0927 22:54:04.730205 (+ 221us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.730210 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.730229 (+ 19us) server_negotiation.cc:373] Connection header received 0927 22:54:04.730277 (+ 48us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.730282 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.730362 (+ 80us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.730451 (+ 89us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.730743 (+ 292us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.730999 (+ 256us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.732200 (+ 1201us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.733787 (+ 1587us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.733883 (+ 96us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.734584 (+ 701us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.734743 (+ 159us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.734946 (+ 203us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.735266 (+ 320us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.735305 (+ 39us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":73} I0927 22:54:04.742517 18725 raft_consensus.cc:1058] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Refusing update from remote peer 00f349d4d9e84c4596252016e4bfc620: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0927 22:54:04.742852 18854 raft_consensus.cc:1058] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Refusing update from remote peer 00f349d4d9e84c4596252016e4bfc620: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0927 22:54:04.742919 18959 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.737909 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.738092 (+ 183us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.738105 (+ 13us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.738187 (+ 82us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.738577 (+ 390us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.738582 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.738595 (+ 13us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.738659 (+ 64us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.738753 (+ 94us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.738762 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.739300 (+ 538us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.739303 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.740029 (+ 726us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.740041 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.741352 (+ 1311us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.741357 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.741575 (+ 218us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.741636 (+ 61us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.742467 (+ 831us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.742477 (+ 10us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.742614 (+ 137us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.742654 (+ 40us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":54} I0927 22:54:04.743422 18945 consensus_queue.cc:962] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0927 22:54:04.743662 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.738229 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40486 0927 22:54:04.738365 (+ 136us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.738368 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.738381 (+ 13us) server_negotiation.cc:373] Connection header received 0927 22:54:04.738413 (+ 32us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.738416 (+ 3us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.738471 (+ 55us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.738546 (+ 75us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.738873 (+ 327us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.739125 (+ 252us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.740196 (+ 1071us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.741116 (+ 920us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.741209 (+ 93us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.741988 (+ 779us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.742150 (+ 162us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.742301 (+ 151us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.743381 (+ 1080us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.743421 (+ 40us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":53} I0927 22:54:04.743906 18938 consensus_queue.cc:962] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0927 22:54:04.768890 18928 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.750318 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:32860 0927 22:54:04.750508 (+ 190us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.750513 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.750753 (+ 240us) server_negotiation.cc:373] Connection header received 0927 22:54:04.751016 (+ 263us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.751022 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.751689 (+ 667us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.751780 (+ 91us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.752271 (+ 491us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.753463 (+ 1192us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.754424 (+ 961us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.755161 (+ 737us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.755262 (+ 101us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.767185 (+ 11923us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.767641 (+ 456us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.767930 (+ 289us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.768524 (+ 594us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.768575 (+ 51us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":53} I0927 22:54:04.769502 18959 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.750020 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.1:40273 0927 22:54:04.750692 (+ 672us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.750708 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.750847 (+ 139us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.751941 (+ 1094us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.751945 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.751957 (+ 12us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.752015 (+ 58us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.752133 (+ 118us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.752144 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.753626 (+ 1482us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.753631 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.754293 (+ 662us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.754304 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.755423 (+ 1119us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.755432 (+ 9us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.755574 (+ 142us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.755640 (+ 66us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.768157 (+ 12517us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.768169 (+ 12us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.769115 (+ 946us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.769174 (+ 59us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":514} I0927 22:54:04.783099 18959 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.778207 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.1:40273 0927 22:54:04.778923 (+ 716us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.778941 (+ 18us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.779080 (+ 139us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.779378 (+ 298us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.779384 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.779399 (+ 15us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.779473 (+ 74us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.779540 (+ 67us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.779548 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.780019 (+ 471us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.780024 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.780697 (+ 673us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.780707 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.781946 (+ 1239us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.781950 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.782042 (+ 92us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.782132 (+ 90us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.782720 (+ 588us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.782730 (+ 10us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.782842 (+ 112us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.782874 (+ 32us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":499} I0927 22:54:04.783282 18928 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.778568 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:32862 0927 22:54:04.778874 (+ 306us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.778880 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.778988 (+ 108us) server_negotiation.cc:373] Connection header received 0927 22:54:04.779189 (+ 201us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.779192 (+ 3us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.779257 (+ 65us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.779338 (+ 81us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.779676 (+ 338us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.779862 (+ 186us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.780833 (+ 971us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.781812 (+ 979us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.781916 (+ 104us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.782373 (+ 457us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.782526 (+ 153us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.782685 (+ 159us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.782936 (+ 251us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.782975 (+ 39us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":147} I0927 22:54:04.800952 18959 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.794299 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:54:04.794534 (+ 235us) negotiation.cc:98] Waiting for socket to connect 0927 22:54:04.794551 (+ 17us) client_negotiation.cc:160] Beginning negotiation 0927 22:54:04.794684 (+ 133us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:54:04.795509 (+ 825us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:54:04.795514 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:54:04.795530 (+ 16us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:54:04.795617 (+ 87us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.795701 (+ 84us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.795712 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.797185 (+ 1473us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.797190 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.797943 (+ 753us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:54:04.797955 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.799332 (+ 1377us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.799339 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:54:04.799473 (+ 134us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.799537 (+ 64us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.800391 (+ 854us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.800403 (+ 12us) client_negotiation.cc:695] Sending connection context 0927 22:54:04.800589 (+ 186us) client_negotiation.cc:225] Negotiation successful 0927 22:54:04.800636 (+ 47us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":73} I0927 22:54:04.801745 18640 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:54:04.794610 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40492 0927 22:54:04.795053 (+ 443us) server_negotiation.cc:176] Beginning negotiation 0927 22:54:04.795078 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0927 22:54:04.795111 (+ 33us) server_negotiation.cc:373] Connection header received 0927 22:54:04.795178 (+ 67us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:54:04.795184 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:54:04.795277 (+ 93us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:54:04.795366 (+ 89us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:54:04.795883 (+ 517us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.796914 (+ 1031us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.798143 (+ 1229us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:54:04.799046 (+ 903us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:54:04.799170 (+ 124us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:54:04.799907 (+ 737us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:54:04.800123 (+ 216us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:54:04.800288 (+ 165us) server_negotiation.cc:901] Waiting for connection context 0927 22:54:04.801420 (+ 1132us) server_negotiation.cc:264] Negotiation successful 0927 22:54:04.801467 (+ 47us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":296} I0927 22:54:04.802292 18323 catalog_manager.cc:1666] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40492: table { table_name: "test-table" } I0927 22:54:04.812417 18323 catalog_manager.cc:3806] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Sending DeleteTablet for 3 replicas of tablet 224a68a39c264e3d85e18d8f840e57ce I0927 22:54:04.812610 18323 catalog_manager.cc:2922] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 224a68a39c264e3d85e18d8f840e57ce on 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2:41325) (Table deleted at 2018-09-27 22:54:04 UTC) I0927 22:54:04.813393 18705 tablet_service.cc:799] Processing DeleteTablet for tablet 224a68a39c264e3d85e18d8f840e57ce with delete_type TABLET_DATA_DELETED (Table deleted at 2018-09-27 22:54:04 UTC) from {username='kudu', principal='kudu/127.17.51.62@KRBTEST.COM'} at 127.0.0.1:40738 I0927 22:54:04.813555 18705 tablet_replica.cc:262] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: stopping tablet replica I0927 22:54:04.813637 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op CompactRowSetsOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.813697 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op MinorDeltaCompactionOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.813745 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op MajorDeltaCompactionOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.813841 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op UndoDeltaBlockGCOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.813899 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op FlushMRSOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.813944 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op FlushDeltaMemStoresOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.813976 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op LogGCOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.814025 18705 raft_consensus.cc:2012] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Raft consensus shutting down. I0927 22:54:04.814170 18705 raft_consensus.cc:2039] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Raft consensus is shut down! I0927 22:54:04.815506 18705 ts_tablet_manager.cc:1277] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: Deleting tablet data with delete state TABLET_DATA_DELETED W0927 22:54:04.828560 18528 consensus_peers.cc:435] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 -> Peer 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2:41325): Couldn't send request to peer 26b5292fbabd4fbe9b8146fa205ecb22 for tablet 224a68a39c264e3d85e18d8f840e57ce. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times. I0927 22:54:04.833765 18705 ts_tablet_manager.cc:1290] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0927 22:54:04.835554 18705 log.cc:981] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: Deleting WAL directory at /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/wal/wals/224a68a39c264e3d85e18d8f840e57ce I0927 22:54:04.835791 18323 catalog_manager.cc:2922] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 224a68a39c264e3d85e18d8f840e57ce on 00f349d4d9e84c4596252016e4bfc620 (127.17.51.1:40273) (Table deleted at 2018-09-27 22:54:04 UTC) I0927 22:54:04.836331 18705 ts_tablet_manager.cc:1310] T 224a68a39c264e3d85e18d8f840e57ce P 26b5292fbabd4fbe9b8146fa205ecb22: Deleting consensus metadata I0927 22:54:04.837406 18323 catalog_manager.cc:2922] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 224a68a39c264e3d85e18d8f840e57ce on 54434d11164442ca96a66cda770a149c (127.17.51.3:44871) (Table deleted at 2018-09-27 22:54:04 UTC) I0927 22:54:04.838157 18834 tablet_service.cc:799] Processing DeleteTablet for tablet 224a68a39c264e3d85e18d8f840e57ce with delete_type TABLET_DATA_DELETED (Table deleted at 2018-09-27 22:54:04 UTC) from {username='kudu', principal='kudu/127.17.51.62@KRBTEST.COM'} at 127.0.0.1:42128 I0927 22:54:04.838310 18834 tablet_replica.cc:262] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: stopping tablet replica I0927 22:54:04.838388 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op CompactRowSetsOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.838452 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op MinorDeltaCompactionOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.838500 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op MajorDeltaCompactionOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.838634 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op UndoDeltaBlockGCOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.838722 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op FlushMRSOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.838785 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op FlushDeltaMemStoresOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.838829 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op LogGCOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.838894 18834 raft_consensus.cc:2012] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Raft consensus shutting down. I0927 22:54:04.839022 18834 raft_consensus.cc:2039] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Raft consensus is shut down! I0927 22:54:04.840268 18834 ts_tablet_manager.cc:1277] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: Deleting tablet data with delete state TABLET_DATA_DELETED I0927 22:54:04.840436 18308 catalog_manager.cc:2900] TS 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2:41325): tablet 224a68a39c264e3d85e18d8f840e57ce (table test-table [id=53149fe3c1184e0e87ab180fa11b73de]) successfully deleted I0927 22:54:04.844637 18574 tablet_service.cc:799] Processing DeleteTablet for tablet 224a68a39c264e3d85e18d8f840e57ce with delete_type TABLET_DATA_DELETED (Table deleted at 2018-09-27 22:54:04 UTC) from {username='kudu', principal='kudu/127.17.51.62@KRBTEST.COM'} at 127.0.0.1:32812 I0927 22:54:04.844815 18574 tablet_replica.cc:262] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: stopping tablet replica I0927 22:54:04.844883 18834 ts_tablet_manager.cc:1290] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0927 22:54:04.844909 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op CompactRowSetsOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.844981 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op MinorDeltaCompactionOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.845049 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op MajorDeltaCompactionOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.845042 18834 log.cc:981] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: Deleting WAL directory at /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/wal/wals/224a68a39c264e3d85e18d8f840e57ce I0927 22:54:04.845141 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op UndoDeltaBlockGCOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.845207 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op FlushMRSOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.845263 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op FlushDeltaMemStoresOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.845304 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op LogGCOp(224a68a39c264e3d85e18d8f840e57ce) I0927 22:54:04.845366 18574 raft_consensus.cc:2012] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 1 LEADER]: Raft consensus shutting down. I0927 22:54:04.845464 18574 consensus_peers.cc:457] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 -> Peer 54434d11164442ca96a66cda770a149c (127.17.51.3:44871): Closing peer: 54434d11164442ca96a66cda770a149c I0927 22:54:04.845549 18574 consensus_peers.cc:457] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 -> Peer 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2:41325): Closing peer: 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:54:04.845772 18834 ts_tablet_manager.cc:1310] T 224a68a39c264e3d85e18d8f840e57ce P 54434d11164442ca96a66cda770a149c: Deleting consensus metadata I0927 22:54:04.845814 18574 raft_consensus.cc:2039] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Raft consensus is shut down! I0927 22:54:04.847172 18309 catalog_manager.cc:2900] TS 54434d11164442ca96a66cda770a149c (127.17.51.3:44871): tablet 224a68a39c264e3d85e18d8f840e57ce (table test-table [id=53149fe3c1184e0e87ab180fa11b73de]) successfully deleted I0927 22:54:04.847270 18574 ts_tablet_manager.cc:1277] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: Deleting tablet data with delete state TABLET_DATA_DELETED I0927 22:54:04.852619 18574 ts_tablet_manager.cc:1290] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0927 22:54:04.852727 18574 log.cc:981] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: Deleting WAL directory at /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/wal/wals/224a68a39c264e3d85e18d8f840e57ce I0927 22:54:04.853197 18574 ts_tablet_manager.cc:1310] T 224a68a39c264e3d85e18d8f840e57ce P 00f349d4d9e84c4596252016e4bfc620: Deleting consensus metadata I0927 22:54:04.854506 18306 catalog_manager.cc:2900] TS 00f349d4d9e84c4596252016e4bfc620 (127.17.51.1:40273): tablet 224a68a39c264e3d85e18d8f840e57ce (table test-table [id=53149fe3c1184e0e87ab180fa11b73de]) successfully deleted W0927 22:56:02.219146 18303 init.cc:188] Kerberos reacquire error: : Runtime error: Reacquire error: unable to login from keytab: Cannot contact any KDC for realm 'KRBTEST.COM' W0927 22:56:02.519748 18368 init.cc:188] Kerberos reacquire error: : Runtime error: Reacquire error: unable to login from keytab: Invalid message type W0927 22:56:02.830935 18433 init.cc:188] Kerberos reacquire error: : Runtime error: Reacquire error: unable to login from keytab: Cannot contact any KDC for realm 'KRBTEST.COM' I0927 22:56:02.999150 18443 maintenance_manager.cc:302] P 9fa6f72caa24460e896f673a01c0fcf2: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033342 I0927 22:56:03.000269 18432 tablet.cc:1153] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Flush: entering stage 1 (old memrowset already frozen for inserts) I0927 22:56:03.000316 18432 compaction.cc:914] Selected 1 rowsets to compact: I0927 22:56:03.000360 18432 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0927 22:56:03.000385 18432 tablet.cc:1155] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Memstore in-memory size: 7916 bytes I0927 22:56:03.000434 18432 tablet.cc:1444] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6300011908316626944 or (T in {6300011908316626944})}] I0927 22:56:03.006211 18432 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0927 22:56:03.016705 18432 tablet.cc:1529] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0927 22:56:03.017015 18432 tablet.cc:1584] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0927 22:56:03.017050 18432 tablet.cc:1586] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6300011908316626944 or (T in {6300011908316626944})}] I0927 22:56:03.020937 18432 tablet.cc:1634] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Flush successful on 6 rows (7218 bytes) I0927 22:56:03.021266 18432 maintenance_manager.cc:492] P 9fa6f72caa24460e896f673a01c0fcf2: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.022s user 0.015s sys 0.006s Metrics: {"cfile_init":1,"data dirs.queue_time_us":221,"data dirs.run_cpu_time_us":180,"data dirs.run_wall_time_us":977,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":624,"lbm_writes_lt_1ms":27,"thread_start_us":688,"threads_started":2,"wal-append.queue_time_us":290} I0927 22:56:03.022083 18443 maintenance_manager.cc:302] P 9fa6f72caa24460e896f673a01c0fcf2: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 761 bytes on disk I0927 22:56:03.022639 18432 maintenance_manager.cc:492] P 9fa6f72caa24460e896f673a01c0fcf2: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4} I0927 22:56:03.156755 18313 maintenance_manager.cc:302] P 133b07c7153a40f0b5e18b033e7982c0: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033396 I0927 22:56:03.157739 18302 tablet.cc:1153] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Flush: entering stage 1 (old memrowset already frozen for inserts) I0927 22:56:03.157786 18302 compaction.cc:914] Selected 1 rowsets to compact: I0927 22:56:03.157829 18302 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0927 22:56:03.157862 18302 tablet.cc:1155] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Memstore in-memory size: 7916 bytes I0927 22:56:03.157908 18302 tablet.cc:1444] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6300011908316626944 or (T in {6300011908316626944})}] I0927 22:56:03.163417 18302 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0927 22:56:03.174775 18302 tablet.cc:1529] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0927 22:56:03.175107 18302 tablet.cc:1584] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0927 22:56:03.175146 18302 tablet.cc:1586] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6300011908316626944 or (T in {6300011908316626944})}] I0927 22:56:03.178242 18302 tablet.cc:1634] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Flush successful on 6 rows (7218 bytes) I0927 22:56:03.178575 18302 maintenance_manager.cc:492] P 133b07c7153a40f0b5e18b033e7982c0: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.014s sys 0.005s Metrics: {"cfile_init":1,"data dirs.queue_time_us":308,"data dirs.run_cpu_time_us":169,"data dirs.run_wall_time_us":1002,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1085,"lbm_writes_lt_1ms":27,"thread_start_us":360,"threads_started":2,"wal-append.queue_time_us":227} I0927 22:56:03.179457 18313 maintenance_manager.cc:302] P 133b07c7153a40f0b5e18b033e7982c0: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 761 bytes on disk I0927 22:56:03.179988 18302 maintenance_manager.cc:492] P 133b07c7153a40f0b5e18b033e7982c0: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4} I0927 22:56:03.206816 18378 maintenance_manager.cc:302] P 93ce3bfa2f2a4d00abf078a11b3b7fec: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033378 I0927 22:56:03.207787 18367 tablet.cc:1153] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Flush: entering stage 1 (old memrowset already frozen for inserts) I0927 22:56:03.207830 18367 compaction.cc:914] Selected 1 rowsets to compact: I0927 22:56:03.207862 18367 compaction.cc:917] memrowset(current size on disk: ~0 bytes) I0927 22:56:03.207875 18367 tablet.cc:1155] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Memstore in-memory size: 7916 bytes I0927 22:56:03.207913 18367 tablet.cc:1444] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6300011908316626944 or (T in {6300011908316626944})}] I0927 22:56:03.213305 18367 multi_column_writer.cc:98] Opened CFile writers for 3 column(s) I0927 22:56:03.239800 18367 tablet.cc:1529] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0927 22:56:03.240146 18367 tablet.cc:1584] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0927 22:56:03.240182 18367 tablet.cc:1586] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6300011908316626944 or (T in {6300011908316626944})}] I0927 22:56:03.245208 18367 tablet.cc:1634] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Flush successful on 6 rows (7218 bytes) I0927 22:56:03.245642 18367 maintenance_manager.cc:492] P 93ce3bfa2f2a4d00abf078a11b3b7fec: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.038s user 0.037s sys 0.000s Metrics: {"cfile_init":1,"data dirs.queue_time_us":333,"data dirs.run_cpu_time_us":195,"data dirs.run_wall_time_us":1063,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":27,"thread_start_us":488,"threads_started":2,"wal-append.queue_time_us":402} I0927 22:56:03.246562 18378 maintenance_manager.cc:302] P 93ce3bfa2f2a4d00abf078a11b3b7fec: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 761 bytes on disk I0927 22:56:03.247146 18367 maintenance_manager.cc:492] P 93ce3bfa2f2a4d00abf078a11b3b7fec: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4} W0927 22:56:03.305065 18523 init.cc:188] Kerberos reacquire error: : Runtime error: Reacquire error: unable to login from keytab: Cannot contact any KDC for realm 'KRBTEST.COM' W0927 22:56:03.765138 18654 init.cc:188] Kerberos reacquire error: : Runtime error: Reacquire error: unable to login from keytab: Cannot contact any KDC for realm 'KRBTEST.COM' W0927 22:56:04.094854 18783 init.cc:188] Kerberos reacquire error: : Runtime error: Reacquire error: unable to login from keytab: Cannot contact any KDC for realm 'KRBTEST.COM' I0927 22:56:04.847210 19070 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:04.842786 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:04.843392 (+ 606us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:04.843411 (+ 19us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:04.843531 (+ 120us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:04.845799 (+ 2268us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:04.845803 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:04.846773 (+ 970us) client_negotiation.cc:430] Kerberos authentication credentials are not available: Not authorized: The referenced credential has expired: Success 0927 22:56:04.846922 (+ 149us) negotiation.cc:304] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.51.62:11080: server requires authentication, but client does not have Kerberos credentials available Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":1102,"threads_started":1} W0927 22:56:04.847694 19073 negotiation.cc:313] Failed RPC negotiation. Trace: 0927 22:56:04.843501 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40514 0927 22:56:04.845318 (+ 1817us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:04.845322 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:04.845378 (+ 56us) server_negotiation.cc:373] Connection header received 0927 22:56:04.845423 (+ 45us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:04.845427 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:04.845502 (+ 75us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:04.845588 (+ 86us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:56:04.847063 (+ 1475us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40514: BlockingRecv error: Recv() got EOF from remote (error 108) Metrics: {"server-negotiator.queue_time_us":1690,"thread_start_us":1576,"threads_started":1} I0927 22:56:04.847883 19071 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:04.842895 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:56:04.845500 (+ 2605us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:04.845517 (+ 17us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:04.845670 (+ 153us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:04.846272 (+ 602us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:04.846278 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:04.847544 (+ 1266us) client_negotiation.cc:430] Kerberos authentication credentials are not available: Not authorized: The referenced credential has expired: Success 0927 22:56:04.847676 (+ 132us) negotiation.cc:304] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.51.61:11081: server requires authentication, but client does not have Kerberos credentials available Metrics: {"client-negotiator.queue_time_us":2452,"thread_start_us":2340,"threads_started":1} W0927 22:56:04.848507 19072 negotiation.cc:313] Failed RPC negotiation. Trace: 0927 22:56:04.843260 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51310 0927 22:56:04.844272 (+ 1012us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:04.844276 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:04.845652 (+ 1376us) server_negotiation.cc:373] Connection header received 0927 22:56:04.845822 (+ 170us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:04.845828 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:04.845943 (+ 115us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:04.846036 (+ 93us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:56:04.848255 (+ 2219us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51310: BlockingRecv error: Recv() got EOF from remote (error 108) Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":797,"threads_started":1} I0927 22:56:04.849220 19070 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:04.846770 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:56:04.847558 (+ 788us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:04.847569 (+ 11us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:04.847643 (+ 74us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:04.848024 (+ 381us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:04.848029 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:04.848977 (+ 948us) client_negotiation.cc:430] Kerberos authentication credentials are not available: Not authorized: The referenced credential has expired: Success 0927 22:56:04.849070 (+ 93us) negotiation.cc:304] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.51.60:11082: server requires authentication, but client does not have Kerberos credentials available Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":814,"threads_started":1} W0927 22:56:04.849356 19074 negotiation.cc:313] Failed RPC negotiation. Trace: 0927 22:56:04.843549 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42608 0927 22:56:04.845978 (+ 2429us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:04.845982 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:04.847727 (+ 1745us) server_negotiation.cc:373] Connection header received 0927 22:56:04.847785 (+ 58us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:04.847789 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:04.847895 (+ 106us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:04.847977 (+ 82us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:56:04.849177 (+ 1200us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42608: BlockingRecv error: Recv() got EOF from remote (error 108) Metrics: {"server-negotiator.queue_time_us":2253,"thread_start_us":2884,"threads_started":1} Sep 27 22:56:04 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): setting up network... Sep 27 22:56:04 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): listening on fd 9: udp 0.0.0.0.53876 (pktinfo) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 53876 Sep 27 22:56:04 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): skipping unrecognized local address family 17 Sep 27 22:56:04 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): skipping unrecognized local address family 17 krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked Sep 27 22:56:04 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): listening on fd 10: udp fe80::42:acff:fe11:2%eth0.53876 Sep 27 22:56:04 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): set up 2 sockets Sep 27 22:56:04 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): commencing operation krb5kdc: starting... W0927 22:56:05.192492 17612 mini_kdc.cc:117] Time spent starting KDC: real 0.341s user 0.000s sys 0.016s Sep 27 22:56:05 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: Sep 27 22:56:05 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.61@KRBTEST.COM Sep 27 22:56:05 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.60@KRBTEST.COM Sep 27 22:56:05 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.62@KRBTEST.COM I0927 22:56:05.242466 19074 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.222170 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42614 0927 22:56:05.222361 (+ 191us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.222365 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.222975 (+ 610us) server_negotiation.cc:373] Connection header received 0927 22:56:05.223212 (+ 237us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.223218 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.223307 (+ 89us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.223409 (+ 102us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:56:05.224985 (+ 1576us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.225199 (+ 214us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.226145 (+ 946us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.226896 (+ 751us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.227005 (+ 109us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.233256 (+ 6251us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:56:05.233292 (+ 36us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:56:05.233297 (+ 5us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:56:05.233339 (+ 42us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:56:05.236608 (+ 3269us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:05.239415 (+ 2807us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:05.239422 (+ 7us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:05.239424 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:05.239733 (+ 309us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:05.240262 (+ 529us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:05.240266 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:05.240267 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:05.240447 (+ 180us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:56:05.240678 (+ 231us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.241481 (+ 803us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.242133 (+ 652us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":57} I0927 22:56:05.242621 19087 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.222014 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:56:05.222906 (+ 892us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.222925 (+ 19us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.223050 (+ 125us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.223465 (+ 415us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.223470 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.224652 (+ 1182us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:56:05.224755 (+ 103us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.224871 (+ 116us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.224883 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.225366 (+ 483us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.225370 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.225988 (+ 618us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.225998 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.227076 (+ 1078us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.227081 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.227189 (+ 108us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.227371 (+ 182us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:56:05.227374 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:56:05.232637 (+ 5263us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:56:05.238871 (+ 6234us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:05.238875 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:05.238877 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:05.239168 (+ 291us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:05.239988 (+ 820us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:05.239992 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:05.239994 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:05.240051 (+ 57us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:05.240660 (+ 609us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:56:05.240664 (+ 4us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:56:05.240889 (+ 225us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.241303 (+ 414us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.242186 (+ 883us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":3007,"thread_start_us":2619,"threads_started":1} I0927 22:56:05.243643 19085 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.221900 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:56:05.222420 (+ 520us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.222435 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.222585 (+ 150us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.223021 (+ 436us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.223025 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.223940 (+ 915us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:56:05.224070 (+ 130us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.224224 (+ 154us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.224237 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.225078 (+ 841us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.225081 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.225620 (+ 539us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.225630 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.226513 (+ 883us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.226517 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.226613 (+ 96us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.226799 (+ 186us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:56:05.226802 (+ 3us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:56:05.230133 (+ 3331us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:56:05.233736 (+ 3603us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:05.233741 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:05.233743 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:05.235879 (+ 2136us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:05.236960 (+ 1081us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:05.236964 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:05.236966 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:05.237022 (+ 56us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:05.240357 (+ 3335us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:56:05.240361 (+ 4us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:56:05.240833 (+ 472us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.241524 (+ 691us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.242304 (+ 780us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":2917,"thread_start_us":436,"threads_started":1} I0927 22:56:05.243934 19072 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.221920 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51316 0927 22:56:05.222235 (+ 315us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.222242 (+ 7us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.222518 (+ 276us) server_negotiation.cc:373] Connection header received 0927 22:56:05.222721 (+ 203us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.222726 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.222820 (+ 94us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.222934 (+ 114us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:56:05.224350 (+ 1416us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.224622 (+ 272us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.225753 (+ 1131us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.226367 (+ 614us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.226472 (+ 105us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.230392 (+ 3920us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:56:05.230424 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:56:05.230429 (+ 5us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:56:05.230469 (+ 40us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:56:05.233428 (+ 2959us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:05.236150 (+ 2722us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:05.236155 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:05.236159 (+ 4us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:05.236543 (+ 384us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:05.237539 (+ 996us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:05.237543 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:05.237545 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:05.239991 (+ 2446us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:56:05.240227 (+ 236us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.243027 (+ 2800us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.243618 (+ 591us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":163} I0927 22:56:05.246599 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.222341 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.224154 (+ 1813us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.224171 (+ 17us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.224764 (+ 593us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.225383 (+ 619us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.225388 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.227479 (+ 2091us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:56:05.227557 (+ 78us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.228369 (+ 812us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.228378 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.228823 (+ 445us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.228828 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.229464 (+ 636us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.229474 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.232275 (+ 2801us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.232282 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.232511 (+ 229us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.232724 (+ 213us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:56:05.232728 (+ 4us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:56:05.235551 (+ 2823us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:56:05.240403 (+ 4852us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:05.240406 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:05.240409 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:05.240946 (+ 537us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:05.244568 (+ 3622us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:05.244573 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:05.244576 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:05.244656 (+ 80us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:05.245778 (+ 1122us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:56:05.245783 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:56:05.245916 (+ 133us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.246095 (+ 179us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.246320 (+ 225us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1624,"mutex_wait_us":131,"thread_start_us":807,"threads_started":1} I0927 22:56:05.246829 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.221824 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40520 0927 22:56:05.221993 (+ 169us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.221997 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.224226 (+ 2229us) server_negotiation.cc:373] Connection header received 0927 22:56:05.224950 (+ 724us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.224955 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.225033 (+ 78us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.225124 (+ 91us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:56:05.228531 (+ 3407us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.228690 (+ 159us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.229597 (+ 907us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.230117 (+ 520us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.230223 (+ 106us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.235818 (+ 5595us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:56:05.235850 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:56:05.235853 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:56:05.235907 (+ 54us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:56:05.238612 (+ 2705us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:05.241171 (+ 2559us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:05.241175 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:05.241177 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:05.241433 (+ 256us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:05.244872 (+ 3439us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:05.244875 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:05.244877 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:05.245075 (+ 198us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:56:05.245284 (+ 209us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.246221 (+ 937us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.246620 (+ 399us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":53} I0927 22:56:05.253242 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.249639 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.249839 (+ 200us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.249851 (+ 12us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.250089 (+ 238us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.250400 (+ 311us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.250403 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.250414 (+ 11us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.250466 (+ 52us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.250514 (+ 48us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.250521 (+ 7us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.250851 (+ 330us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.250854 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.251385 (+ 531us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.251394 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.252199 (+ 805us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.252203 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.252288 (+ 85us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.252328 (+ 40us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.252890 (+ 562us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.252898 (+ 8us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.253007 (+ 109us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.253036 (+ 29us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":67} I0927 22:56:05.253314 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.249867 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40526 0927 22:56:05.250005 (+ 138us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.250009 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.250023 (+ 14us) server_negotiation.cc:373] Connection header received 0927 22:56:05.250215 (+ 192us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.250220 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.250300 (+ 80us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.250382 (+ 82us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.250620 (+ 238us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.250758 (+ 138us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.251534 (+ 776us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.252038 (+ 504us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.252131 (+ 93us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.252574 (+ 443us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.252700 (+ 126us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.252819 (+ 119us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.253081 (+ 262us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.253117 (+ 36us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":40} I0927 22:56:05.253885 18323 catalog_manager.cc:1294] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40526: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "val" type: INT32 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0927 22:56:05.254436 18323 catalog_manager.cc:1450] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0927 22:56:05.307049 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.278964 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.279239 (+ 275us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.279257 (+ 18us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.279403 (+ 146us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.279753 (+ 350us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.279757 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.279769 (+ 12us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.279831 (+ 62us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.279909 (+ 78us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.279918 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.280754 (+ 836us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.280757 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.281304 (+ 547us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.281314 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.294969 (+ 13655us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.294980 (+ 11us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.304701 (+ 9721us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.304817 (+ 116us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.306498 (+ 1681us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.306509 (+ 11us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.306643 (+ 134us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.306693 (+ 50us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":91} I0927 22:56:05.307735 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.278992 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40528 0927 22:56:05.279194 (+ 202us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.279200 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.279330 (+ 130us) server_negotiation.cc:373] Connection header received 0927 22:56:05.279538 (+ 208us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.279544 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.279635 (+ 91us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.279741 (+ 106us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.280043 (+ 302us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.280226 (+ 183us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.281442 (+ 1216us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.282929 (+ 1487us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.283029 (+ 100us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.305940 (+ 22911us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.306156 (+ 216us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.306315 (+ 159us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.307492 (+ 1177us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.307552 (+ 60us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":73} I0927 22:56:05.318029 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.313021 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.313272 (+ 251us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.313290 (+ 18us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.313401 (+ 111us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.314069 (+ 668us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.314075 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.314103 (+ 28us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.314175 (+ 72us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.314244 (+ 69us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.314256 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.314763 (+ 507us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.314767 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.315260 (+ 493us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.315270 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.316202 (+ 932us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.316206 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.316299 (+ 93us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.316345 (+ 46us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.317684 (+ 1339us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.317692 (+ 8us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.317787 (+ 95us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.317819 (+ 32us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":82} I0927 22:56:05.319126 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.313566 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40536 0927 22:56:05.313773 (+ 207us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.313778 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.313793 (+ 15us) server_negotiation.cc:373] Connection header received 0927 22:56:05.313837 (+ 44us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.313842 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.313920 (+ 78us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.313992 (+ 72us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.314402 (+ 410us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.314601 (+ 199us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.315384 (+ 783us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.315906 (+ 522us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.315995 (+ 89us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.316596 (+ 601us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.316710 (+ 114us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.316808 (+ 98us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.318861 (+ 2053us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.318901 (+ 40us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":106} I0927 22:56:05.319217 19099 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.282875 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.3:44871 0927 22:56:05.309806 (+ 26931us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.309819 (+ 13us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.309900 (+ 81us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.313367 (+ 3467us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.313372 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.313402 (+ 30us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:56:05.313502 (+ 100us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.313560 (+ 58us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.313571 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.315160 (+ 1589us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.315165 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.316324 (+ 1159us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.316338 (+ 14us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.318483 (+ 2145us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.318490 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.318640 (+ 150us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.318741 (+ 101us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.318954 (+ 213us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.318996 (+ 42us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":26833,"mutex_wait_us":18,"thread_start_us":26360,"threads_started":1} I0927 22:56:05.319347 19102 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.283168 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42224 0927 22:56:05.283593 (+ 425us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.283597 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.312506 (+ 28909us) server_negotiation.cc:373] Connection header received 0927 22:56:05.312575 (+ 69us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.312580 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.312668 (+ 88us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.312755 (+ 87us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:56:05.313711 (+ 956us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.313918 (+ 207us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.316540 (+ 2622us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.317404 (+ 864us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.317475 (+ 71us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.317554 (+ 79us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.319050 (+ 1496us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.319101 (+ 51us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":179,"threads_started":1} I0927 22:56:05.319901 19100 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.282738 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.2:41325 0927 22:56:05.309440 (+ 26702us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.309452 (+ 12us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.309535 (+ 83us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.314941 (+ 5406us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.314946 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.314973 (+ 27us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:56:05.315082 (+ 109us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.315136 (+ 54us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.315146 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.315648 (+ 502us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.315653 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.316932 (+ 1279us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.316944 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.318850 (+ 1906us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.318856 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.319506 (+ 650us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.319565 (+ 59us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.319716 (+ 151us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.319747 (+ 31us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":26592,"mutex_wait_us":575,"thread_start_us":26909,"threads_started":1} I0927 22:56:05.320087 19103 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.289269 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.1:40273 0927 22:56:05.313138 (+ 23869us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.313152 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.313243 (+ 91us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.314493 (+ 1250us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.314498 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.314526 (+ 28us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:56:05.314704 (+ 178us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.314754 (+ 50us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.314764 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.316101 (+ 1337us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.316104 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.317479 (+ 1375us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.317488 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.318674 (+ 1186us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.318679 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.318850 (+ 171us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.319499 (+ 649us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.319758 (+ 259us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.319848 (+ 90us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":23754,"mutex_wait_us":703,"thread_start_us":23683,"threads_started":1} I0927 22:56:05.320283 18834 tablet_service.cc:758] Processing CreateTablet for tablet dfb9184aa58e4d0d9661f5db4fb87392 (table=test-table [id=6e2a53728abc48769485469b318e9273]), partition=RANGE (key) PARTITION UNBOUNDED I0927 22:56:05.320729 18834 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet dfb9184aa58e4d0d9661f5db4fb87392. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:56:05.320745 19101 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.283042 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40830 0927 22:56:05.314633 (+ 31591us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.314645 (+ 12us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.314670 (+ 25us) server_negotiation.cc:373] Connection header received 0927 22:56:05.314737 (+ 67us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.314743 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.314845 (+ 102us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.314935 (+ 90us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:56:05.315279 (+ 344us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.315526 (+ 247us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.317091 (+ 1565us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.318550 (+ 1459us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.318910 (+ 360us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.319015 (+ 105us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.320317 (+ 1302us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.320382 (+ 65us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":25259,"thread_start_us":25097,"threads_started":1} I0927 22:56:05.321460 18705 tablet_service.cc:758] Processing CreateTablet for tablet dfb9184aa58e4d0d9661f5db4fb87392 (table=test-table [id=6e2a53728abc48769485469b318e9273]), partition=RANGE (key) PARTITION UNBOUNDED I0927 22:56:05.321801 18705 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet dfb9184aa58e4d0d9661f5db4fb87392. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:56:05.324585 18834 ts_tablet_manager.cc:1080] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: Registered tablet (data state: TABLET_DATA_READY) I0927 22:56:05.324996 18705 ts_tablet_manager.cc:1080] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Registered tablet (data state: TABLET_DATA_READY) I0927 22:56:05.326184 19105 ts_tablet_manager.cc:938] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: Bootstrapping tablet I0927 22:56:05.326294 19105 tablet_bootstrap.cc:436] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: Bootstrap starting. I0927 22:56:05.327136 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.322849 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.323602 (+ 753us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.323615 (+ 13us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.323721 (+ 106us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.324091 (+ 370us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.324094 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.324106 (+ 12us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.324167 (+ 61us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.324219 (+ 52us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.324226 (+ 7us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.324593 (+ 367us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.324596 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.325107 (+ 511us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.325116 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.325901 (+ 785us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.325905 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.326045 (+ 140us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.326110 (+ 65us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.326685 (+ 575us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.326694 (+ 9us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.326806 (+ 112us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.326855 (+ 49us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":604} I0927 22:56:05.327324 19105 tablet_bootstrap.cc:581] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: No blocks or log segments found. Creating new log. I0927 22:56:05.327596 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.323284 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40538 0927 22:56:05.323438 (+ 154us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.323441 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.323803 (+ 362us) server_negotiation.cc:373] Connection header received 0927 22:56:05.323848 (+ 45us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.323851 (+ 3us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.323921 (+ 70us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.323975 (+ 54us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.324332 (+ 357us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.324485 (+ 153us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.325231 (+ 746us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.325723 (+ 492us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.325788 (+ 65us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.326370 (+ 582us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.326475 (+ 105us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.326574 (+ 99us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.327438 (+ 864us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.327466 (+ 28us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":40} I0927 22:56:05.328537 19106 ts_tablet_manager.cc:938] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Bootstrapping tablet I0927 22:56:05.328662 19106 tablet_bootstrap.cc:436] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Bootstrap starting. I0927 22:56:05.329823 19106 tablet_bootstrap.cc:581] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: No blocks or log segments found. Creating new log. I0927 22:56:05.333307 19104 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.289447 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:32906 0927 22:56:05.307092 (+ 17645us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.307098 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.314106 (+ 7008us) server_negotiation.cc:373] Connection header received 0927 22:56:05.314289 (+ 183us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.314296 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.314388 (+ 92us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.314480 (+ 92us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:56:05.314924 (+ 444us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.315225 (+ 301us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.317605 (+ 2380us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.318518 (+ 913us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.318608 (+ 90us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.318693 (+ 85us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.330631 (+ 11938us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.330899 (+ 268us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":17457,"thread_start_us":17630,"threads_started":1} I0927 22:56:05.333930 18574 tablet_service.cc:758] Processing CreateTablet for tablet dfb9184aa58e4d0d9661f5db4fb87392 (table=test-table [id=6e2a53728abc48769485469b318e9273]), partition=RANGE (key) PARTITION UNBOUNDED I0927 22:56:05.334326 18574 data_dirs.cc:931] Could only allocate 1 dirs of requested 3 for tablet dfb9184aa58e4d0d9661f5db4fb87392. 1 dirs total, 0 dirs full, 0 dirs failed I0927 22:56:05.335292 19106 tablet_bootstrap.cc:436] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: No bootstrap required, opened a new log I0927 22:56:05.335353 19105 tablet_bootstrap.cc:436] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: No bootstrap required, opened a new log I0927 22:56:05.335392 19106 ts_tablet_manager.cc:955] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s I0927 22:56:05.335549 19105 ts_tablet_manager.cc:955] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s I0927 22:56:05.335924 19106 raft_consensus.cc:304] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:56:05.336098 19106 raft_consensus.cc:330] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:56:05.336132 19106 raft_consensus.cc:605] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26b5292fbabd4fbe9b8146fa205ecb22, State: Initialized, Role: FOLLOWER I0927 22:56:05.336313 19105 raft_consensus.cc:304] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:56:05.336549 19105 raft_consensus.cc:330] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:56:05.336822 19105 raft_consensus.cc:605] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54434d11164442ca96a66cda770a149c, State: Initialized, Role: FOLLOWER I0927 22:56:05.337047 19105 consensus_queue.cc:226] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:56:05.337594 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.331664 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.332263 (+ 599us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.332292 (+ 29us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.332392 (+ 100us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.332911 (+ 519us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.332917 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.332931 (+ 14us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.333022 (+ 91us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.333089 (+ 67us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.333099 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.333461 (+ 362us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.333465 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.333945 (+ 480us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.333955 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.335279 (+ 1324us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.335286 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.335536 (+ 250us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.335663 (+ 127us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.336908 (+ 1245us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.336920 (+ 12us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.337152 (+ 232us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.337293 (+ 141us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":418} I0927 22:56:05.338469 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.331746 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40540 0927 22:56:05.332648 (+ 902us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.332652 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.332665 (+ 13us) server_negotiation.cc:373] Connection header received 0927 22:56:05.332700 (+ 35us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.332703 (+ 3us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.332762 (+ 59us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.332854 (+ 92us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.333214 (+ 360us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.333332 (+ 118us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.334166 (+ 834us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.334976 (+ 810us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.335102 (+ 126us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.336005 (+ 903us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.336359 (+ 354us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.336749 (+ 390us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.338184 (+ 1435us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.338230 (+ 46us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":812} I0927 22:56:05.338904 18574 ts_tablet_manager.cc:1080] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: Registered tablet (data state: TABLET_DATA_READY) I0927 22:56:05.336293 19106 consensus_queue.cc:226] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:56:05.339442 19110 ts_tablet_manager.cc:938] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: Bootstrapping tablet I0927 22:56:05.339552 19110 tablet_bootstrap.cc:436] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: Bootstrap starting. I0927 22:56:05.340992 19105 ts_tablet_manager.cc:983] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s I0927 22:56:05.343122 19110 tablet_bootstrap.cc:581] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: No blocks or log segments found. Creating new log. I0927 22:56:05.344745 19111 raft_consensus.cc:436] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0927 22:56:05.344833 19111 raft_consensus.cc:2700] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Snoozing failure detection for 0.086s (starting election) I0927 22:56:05.344926 19111 raft_consensus.cc:458] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:56:05.345593 19106 ts_tablet_manager.cc:983] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s I0927 22:56:05.346406 19111 leader_election.cc:231] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 00f349d4d9e84c4596252016e4bfc620 I0927 22:56:05.346462 19110 tablet_bootstrap.cc:436] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: No bootstrap required, opened a new log I0927 22:56:05.346571 19110 ts_tablet_manager.cc:955] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s I0927 22:56:05.347038 19109 raft_consensus.cc:436] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0927 22:56:05.347121 19109 raft_consensus.cc:2700] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: Snoozing failure detection for 0.093s (starting election) I0927 22:56:05.347183 19109 raft_consensus.cc:458] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:56:05.347789 19109 leader_election.cc:231] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 00f349d4d9e84c4596252016e4bfc620 I0927 22:56:05.352157 19111 leader_election.cc:231] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 54434d11164442ca96a66cda770a149c I0927 22:56:05.353229 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.342573 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.348146 (+ 5573us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.348165 (+ 19us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.348305 (+ 140us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.348840 (+ 535us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.348845 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.348859 (+ 14us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.349002 (+ 143us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.349107 (+ 105us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.349120 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.350100 (+ 980us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.350105 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.351048 (+ 943us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.351071 (+ 23us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.351850 (+ 779us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.351857 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.351983 (+ 126us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.352032 (+ 49us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.352708 (+ 676us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.352717 (+ 9us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.352847 (+ 130us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.352893 (+ 46us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":5370} I0927 22:56:05.353236 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.342806 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40542 0927 22:56:05.343033 (+ 227us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.343039 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.348422 (+ 5383us) server_negotiation.cc:373] Connection header received 0927 22:56:05.348512 (+ 90us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.348517 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.348613 (+ 96us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.348695 (+ 82us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.349492 (+ 797us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.349932 (+ 440us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.351195 (+ 1263us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.351697 (+ 502us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.351802 (+ 105us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.352350 (+ 548us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.352493 (+ 143us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.352638 (+ 145us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.352942 (+ 304us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.352989 (+ 47us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":104} I0927 22:56:05.357942 19109 leader_election.cc:231] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:56:05.363646 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.358098 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.359699 (+ 1601us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.359714 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.359825 (+ 111us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.360387 (+ 562us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.360392 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.360405 (+ 13us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.360508 (+ 103us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.360582 (+ 74us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.360593 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.361028 (+ 435us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.361033 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.361659 (+ 626us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.361672 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.362496 (+ 824us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.362503 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.362604 (+ 101us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.362648 (+ 44us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.363198 (+ 550us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.363207 (+ 9us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.363325 (+ 118us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.363366 (+ 41us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1453} I0927 22:56:05.363701 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.358115 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40544 0927 22:56:05.359139 (+ 1024us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.359144 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.359936 (+ 792us) server_negotiation.cc:373] Connection header received 0927 22:56:05.360086 (+ 150us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.360092 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.360176 (+ 84us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.360254 (+ 78us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.360727 (+ 473us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.360921 (+ 194us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.361827 (+ 906us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.362346 (+ 519us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.362440 (+ 94us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.362881 (+ 441us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.362992 (+ 111us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.363138 (+ 146us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.363431 (+ 293us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.363473 (+ 42us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":881} I0927 22:56:05.374155 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.369652 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.369893 (+ 241us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.369904 (+ 11us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.369975 (+ 71us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.370458 (+ 483us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.370463 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.370476 (+ 13us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.370541 (+ 65us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.370606 (+ 65us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.370613 (+ 7us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.370984 (+ 371us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.370989 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.371455 (+ 466us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.371464 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.372555 (+ 1091us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.372562 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.372693 (+ 131us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.372758 (+ 65us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.373616 (+ 858us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.373628 (+ 12us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.373764 (+ 136us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.373827 (+ 63us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":129} I0927 22:56:05.374754 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.369674 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40546 0927 22:56:05.370181 (+ 507us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.370186 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.370204 (+ 18us) server_negotiation.cc:373] Connection header received 0927 22:56:05.370253 (+ 49us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.370258 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.370342 (+ 84us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.370436 (+ 94us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.370732 (+ 296us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.370872 (+ 140us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.371585 (+ 713us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.372268 (+ 683us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.372361 (+ 93us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.373174 (+ 813us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.373313 (+ 139us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.373457 (+ 144us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.374507 (+ 1050us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.374549 (+ 42us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":371} I0927 22:56:05.375644 19110 raft_consensus.cc:304] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:56:05.375836 19110 raft_consensus.cc:330] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0927 22:56:05.375876 19110 raft_consensus.cc:605] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f349d4d9e84c4596252016e4bfc620, State: Initialized, Role: FOLLOWER I0927 22:56:05.376024 19110 consensus_queue.cc:226] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:56:05.381615 19104 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.347178 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.2:47787 0927 22:56:05.347391 (+ 213us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.347395 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.377493 (+ 30098us) server_negotiation.cc:373] Connection header received 0927 22:56:05.377711 (+ 218us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.377716 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.377788 (+ 72us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.377843 (+ 55us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:56:05.378357 (+ 514us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.378521 (+ 164us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.379813 (+ 1292us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.380693 (+ 880us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.380756 (+ 63us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.380823 (+ 67us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.381399 (+ 576us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.381435 (+ 36us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":64} I0927 22:56:05.382118 19113 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.346835 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.1:40273 0927 22:56:05.377420 (+ 30585us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.377439 (+ 19us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.377588 (+ 149us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.377993 (+ 405us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.377997 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.378020 (+ 23us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:56:05.378156 (+ 136us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.378247 (+ 91us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.378256 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.378636 (+ 380us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.378639 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.379669 (+ 1030us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.379679 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.380966 (+ 1287us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.380970 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.381101 (+ 131us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.381140 (+ 39us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.381792 (+ 652us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.381826 (+ 34us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":30435,"mutex_wait_us":2,"thread_start_us":30288,"threads_started":1} I0927 22:56:05.386723 18594 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dfb9184aa58e4d0d9661f5db4fb87392" candidate_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f349d4d9e84c4596252016e4bfc620" is_pre_election: true I0927 22:56:05.386945 19104 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.348399 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.3:54903 0927 22:56:05.381757 (+ 33358us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.381760 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.382804 (+ 1044us) server_negotiation.cc:373] Connection header received 0927 22:56:05.383007 (+ 203us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.383011 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.383075 (+ 64us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.383118 (+ 43us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:56:05.383543 (+ 425us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.383673 (+ 130us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.385270 (+ 1597us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.386049 (+ 779us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.386128 (+ 79us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.386180 (+ 52us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.386693 (+ 513us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.386730 (+ 37us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":33275,"threads_started":1} I0927 22:56:05.386984 18594 raft_consensus.cc:2700] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: Snoozing failure detection for 0.109s (vote granted) I0927 22:56:05.387084 18594 raft_consensus.cc:2215] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26b5292fbabd4fbe9b8146fa205ecb22 in term 0. I0927 22:56:05.387318 19114 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.348152 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.1:40273 0927 22:56:05.382747 (+ 34595us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.382761 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.382897 (+ 136us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.383254 (+ 357us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.383258 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.383281 (+ 23us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:56:05.383372 (+ 91us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.383433 (+ 61us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.383442 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.383782 (+ 340us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.383785 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.385128 (+ 1343us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.385138 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.386304 (+ 1166us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.386309 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.386405 (+ 96us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.386456 (+ 51us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.386823 (+ 367us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.387066 (+ 243us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":34454,"threads_started":1} I0927 22:56:05.387532 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.381431 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.381980 (+ 549us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.381996 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.382191 (+ 195us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.383339 (+ 1148us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.383344 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.383366 (+ 22us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.383431 (+ 65us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.383499 (+ 68us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.383510 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.384008 (+ 498us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.384013 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.384740 (+ 727us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.384753 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.385939 (+ 1186us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.385946 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.386094 (+ 148us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.386160 (+ 66us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.386994 (+ 834us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.387005 (+ 11us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.387197 (+ 192us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.387244 (+ 47us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":395} I0927 22:56:05.388217 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.382685 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40548 0927 22:56:05.382934 (+ 249us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.382939 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.382958 (+ 19us) server_negotiation.cc:373] Connection header received 0927 22:56:05.383009 (+ 51us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.383014 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.383118 (+ 104us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.383200 (+ 82us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.383649 (+ 449us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.383864 (+ 215us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.384916 (+ 1052us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.385678 (+ 762us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.385783 (+ 105us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.386515 (+ 732us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.386664 (+ 149us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.386826 (+ 162us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.387896 (+ 1070us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.387930 (+ 34us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":70} I0927 22:56:05.390542 19110 ts_tablet_manager.cc:983] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: Time spent starting tablet: real 0.044s user 0.000s sys 0.003s I0927 22:56:05.392029 18657 leader_election.cc:363] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 00f349d4d9e84c4596252016e4bfc620 I0927 22:56:05.392145 18657 leader_election.cc:258] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0927 22:56:05.392544 19111 raft_consensus.cc:2700] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Snoozing failure detection for 0.097s (election complete) I0927 22:56:05.392663 19111 raft_consensus.cc:2507] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0927 22:56:05.392720 19111 raft_consensus.cc:436] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0927 22:56:05.392787 19111 raft_consensus.cc:2700] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Snoozing failure detection for 0.095s (starting election) I0927 22:56:05.392843 19111 raft_consensus.cc:2752] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 0 FOLLOWER]: Advancing to term 1 I0927 22:56:05.392973 18594 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dfb9184aa58e4d0d9661f5db4fb87392" candidate_uuid: "54434d11164442ca96a66cda770a149c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f349d4d9e84c4596252016e4bfc620" is_pre_election: true I0927 22:56:05.393191 18594 raft_consensus.cc:2700] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: Snoozing failure detection for 0.087s (vote granted) I0927 22:56:05.393244 18594 raft_consensus.cc:2215] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54434d11164442ca96a66cda770a149c in term 0. I0927 22:56:05.394901 18786 leader_election.cc:363] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 00f349d4d9e84c4596252016e4bfc620 I0927 22:56:05.394963 18786 leader_election.cc:258] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0927 22:56:05.395148 19101 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.387877 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.3:53451 0927 22:56:05.388118 (+ 241us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.388123 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.388141 (+ 18us) server_negotiation.cc:373] Connection header received 0927 22:56:05.388192 (+ 51us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.388197 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.388285 (+ 88us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.388370 (+ 85us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:56:05.390932 (+ 2562us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.391130 (+ 198us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.393257 (+ 2127us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.394201 (+ 944us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.394272 (+ 71us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.394339 (+ 67us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.394889 (+ 550us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.394924 (+ 35us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":71} I0927 22:56:05.395151 19111 raft_consensus.cc:458] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:56:05.395390 19102 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.388214 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.17.51.2:52109 0927 22:56:05.388527 (+ 313us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.388532 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.388553 (+ 21us) server_negotiation.cc:373] Connection header received 0927 22:56:05.388609 (+ 56us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.388613 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.388692 (+ 79us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.388777 (+ 85us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:56:05.390037 (+ 1260us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.390216 (+ 179us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.393700 (+ 3484us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.394532 (+ 832us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.394630 (+ 98us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.394713 (+ 83us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.395134 (+ 421us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.395171 (+ 37us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":88} I0927 22:56:05.395462 19113 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.352580 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.3:44871 0927 22:56:05.382288 (+ 29708us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.382299 (+ 11us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.382362 (+ 63us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.389542 (+ 7180us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.389547 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.389576 (+ 29us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:56:05.389646 (+ 70us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.389715 (+ 69us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.389727 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.392163 (+ 2436us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.392168 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.393592 (+ 1424us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.393604 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.394701 (+ 1097us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.394707 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.394855 (+ 148us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.394901 (+ 46us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.395067 (+ 166us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.395107 (+ 40us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":29618,"mutex_wait_us":1,"thread_start_us":35086,"threads_started":1} I0927 22:56:05.395640 19114 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.358281 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.2:41325 0927 22:56:05.387445 (+ 29164us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.387456 (+ 11us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.387518 (+ 62us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.390675 (+ 3157us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.390680 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.390705 (+ 25us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:56:05.390760 (+ 55us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.390805 (+ 45us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.390811 (+ 6us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.391261 (+ 450us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.391264 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.393100 (+ 1836us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.393110 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.394442 (+ 1332us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.394446 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.394560 (+ 114us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.394596 (+ 36us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.395359 (+ 763us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.395397 (+ 38us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":29082,"thread_start_us":30457,"threads_started":1} I0927 22:56:05.395671 19109 raft_consensus.cc:2700] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: Snoozing failure detection for 0.096s (election complete) I0927 22:56:05.395748 19109 raft_consensus.cc:2507] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0927 22:56:05.395774 19111 leader_election.cc:231] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [CANDIDATE]: Term 1 election: Requesting vote from peer 00f349d4d9e84c4596252016e4bfc620 I0927 22:56:05.395792 19109 raft_consensus.cc:436] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0927 22:56:05.395836 19109 raft_consensus.cc:2700] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: Snoozing failure detection for 0.096s (starting election) I0927 22:56:05.395874 19109 raft_consensus.cc:2752] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 0 FOLLOWER]: Advancing to term 1 I0927 22:56:05.396013 19111 leader_election.cc:231] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [CANDIDATE]: Term 1 election: Requesting vote from peer 54434d11164442ca96a66cda770a149c I0927 22:56:05.396311 18725 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dfb9184aa58e4d0d9661f5db4fb87392" candidate_uuid: "54434d11164442ca96a66cda770a149c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" is_pre_election: true I0927 22:56:05.396533 18725 raft_consensus.cc:2145] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54434d11164442ca96a66cda770a149c in current term 1: Already voted for candidate 26b5292fbabd4fbe9b8146fa205ecb22 in this term. I0927 22:56:05.397205 18788 leader_election.cc:377] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 pre-election: Vote denied by peer 26b5292fbabd4fbe9b8146fa205ecb22. Message: Invalid argument: T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54434d11164442ca96a66cda770a149c in current term 1: Already voted for candidate 26b5292fbabd4fbe9b8146fa205ecb22 in this term. I0927 22:56:05.397328 18594 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dfb9184aa58e4d0d9661f5db4fb87392" candidate_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f349d4d9e84c4596252016e4bfc620" I0927 22:56:05.397477 18594 raft_consensus.cc:2752] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [term 0 FOLLOWER]: Advancing to term 1 I0927 22:56:05.397583 18594 raft_consensus.cc:2700] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: Snoozing failure detection for 0.082s (vote granted) I0927 22:56:05.397559 19109 raft_consensus.cc:458] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:56:05.398003 19109 leader_election.cc:231] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 election: Requesting vote from peer 00f349d4d9e84c4596252016e4bfc620 I0927 22:56:05.398155 19109 leader_election.cc:231] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 election: Requesting vote from peer 26b5292fbabd4fbe9b8146fa205ecb22 I0927 22:56:05.399705 18854 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dfb9184aa58e4d0d9661f5db4fb87392" candidate_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54434d11164442ca96a66cda770a149c" is_pre_election: true I0927 22:56:05.399787 18593 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dfb9184aa58e4d0d9661f5db4fb87392" candidate_uuid: "54434d11164442ca96a66cda770a149c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f349d4d9e84c4596252016e4bfc620" I0927 22:56:05.399880 18854 raft_consensus.cc:2145] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b5292fbabd4fbe9b8146fa205ecb22 in current term 1: Already voted for candidate 54434d11164442ca96a66cda770a149c in this term. I0927 22:56:05.400439 18854 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dfb9184aa58e4d0d9661f5db4fb87392" candidate_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54434d11164442ca96a66cda770a149c" I0927 22:56:05.400621 18854 raft_consensus.cc:2145] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b5292fbabd4fbe9b8146fa205ecb22 in current term 1: Already voted for candidate 54434d11164442ca96a66cda770a149c in this term. I0927 22:56:05.401410 18660 leader_election.cc:377] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 54434d11164442ca96a66cda770a149c. Message: Invalid argument: T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26b5292fbabd4fbe9b8146fa205ecb22 in current term 1: Already voted for candidate 54434d11164442ca96a66cda770a149c in this term. I0927 22:56:05.401441 18725 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "dfb9184aa58e4d0d9661f5db4fb87392" candidate_uuid: "54434d11164442ca96a66cda770a149c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" I0927 22:56:05.401526 18594 raft_consensus.cc:2215] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26b5292fbabd4fbe9b8146fa205ecb22 in term 1. I0927 22:56:05.401624 18725 raft_consensus.cc:2145] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 54434d11164442ca96a66cda770a149c in current term 1: Already voted for candidate 26b5292fbabd4fbe9b8146fa205ecb22 in this term. I0927 22:56:05.401657 18593 raft_consensus.cc:2190] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 54434d11164442ca96a66cda770a149c for term 1 because replica is already servicing an update from a current leader or another vote. I0927 22:56:05.402171 18660 leader_election.cc:377] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [CANDIDATE]: Term 1 election: Vote denied by peer 54434d11164442ca96a66cda770a149c. Message: Invalid argument: T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 26b5292fbabd4fbe9b8146fa205ecb22 in current term 1: Already voted for candidate 54434d11164442ca96a66cda770a149c in this term. I0927 22:56:05.402488 18786 leader_election.cc:377] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 election: Vote denied by peer 00f349d4d9e84c4596252016e4bfc620. Message: Service unavailable: T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 54434d11164442ca96a66cda770a149c for term 1 because replica is already servicing an update from a current leader or another vote. I0927 22:56:05.402592 18788 leader_election.cc:377] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 election: Vote denied by peer 26b5292fbabd4fbe9b8146fa205ecb22. Message: Invalid argument: T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 54434d11164442ca96a66cda770a149c in current term 1: Already voted for candidate 26b5292fbabd4fbe9b8146fa205ecb22 in this term. I0927 22:56:05.402642 18788 leader_election.cc:258] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I0927 22:56:05.402859 19109 raft_consensus.cc:2700] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: Snoozing failure detection for 0.085s (election complete) I0927 22:56:05.402940 19109 raft_consensus.cc:2455] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0927 22:56:05.402962 18657 leader_election.cc:363] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [CANDIDATE]: Term 1 election: Vote granted by peer 00f349d4d9e84c4596252016e4bfc620 I0927 22:56:05.403019 18657 leader_election.cc:258] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0927 22:56:05.403280 19111 raft_consensus.cc:2700] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Snoozing failure detection for 0.111s (election complete) I0927 22:56:05.403396 19111 raft_consensus.cc:2507] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Leader election won for term 1 I0927 22:56:05.406251 19111 raft_consensus.cc:572] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 LEADER]: Becoming Leader. State: Replica: 26b5292fbabd4fbe9b8146fa205ecb22, State: Running, Role: LEADER I0927 22:56:05.406517 19111 consensus_queue.cc:203] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } } I0927 22:56:05.407631 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.398028 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.398245 (+ 217us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.398258 (+ 13us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.398345 (+ 87us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.399211 (+ 866us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.399216 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.399229 (+ 13us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.400209 (+ 980us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.400330 (+ 121us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.400343 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.402762 (+ 2419us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.402767 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.404096 (+ 1329us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.404107 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.405309 (+ 1202us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.405316 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.406127 (+ 811us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.406198 (+ 71us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.407100 (+ 902us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.407116 (+ 16us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.407248 (+ 132us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.407303 (+ 55us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":52} I0927 22:56:05.407655 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.398315 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40550 0927 22:56:05.398772 (+ 457us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.398779 (+ 7us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.398800 (+ 21us) server_negotiation.cc:373] Connection header received 0927 22:56:05.398865 (+ 65us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.398870 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.398967 (+ 97us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.399051 (+ 84us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.400511 (+ 1460us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.400700 (+ 189us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.404232 (+ 3532us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.405042 (+ 810us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.405152 (+ 110us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.406575 (+ 1423us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.406809 (+ 234us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.406970 (+ 161us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.407367 (+ 397us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.407410 (+ 43us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":309} I0927 22:56:05.409834 18324 catalog_manager.cc:3595] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 reported cstate change: term changed from 0 to 1, leader changed from to 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2). New cstate: current_term: 1 leader_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26b5292fbabd4fbe9b8146fa205ecb22" member_type: VOTER last_known_addr { host: "127.17.51.2" port: 41325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 } health_report { overall_health: UNKNOWN } } } W0927 22:56:05.414083 18771 tablet.cc:1679] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Can't schedule compaction. Clean time has not been advanced past its initial value. I0927 22:56:05.423295 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.416970 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40552 0927 22:56:05.418100 (+ 1130us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.418106 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.418713 (+ 607us) server_negotiation.cc:373] Connection header received 0927 22:56:05.418782 (+ 69us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.418788 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.418881 (+ 93us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.418982 (+ 101us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.419305 (+ 323us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.419443 (+ 138us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.420237 (+ 794us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.420773 (+ 536us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.420856 (+ 83us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.422569 (+ 1713us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.422682 (+ 113us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.422797 (+ 115us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.423080 (+ 283us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.423113 (+ 33us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":980} I0927 22:56:05.423297 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.416905 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.418466 (+ 1561us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.418485 (+ 19us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.418607 (+ 122us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.419014 (+ 407us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.419018 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.419030 (+ 12us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.419135 (+ 105us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.419187 (+ 52us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.419195 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.419542 (+ 347us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.419545 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.420082 (+ 537us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.420091 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.420892 (+ 801us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.420895 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.422298 (+ 1403us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.422358 (+ 60us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.422852 (+ 494us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.422860 (+ 8us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.422985 (+ 125us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.423019 (+ 34us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":151} I0927 22:56:05.428067 18594 raft_consensus.cc:1058] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Refusing update from remote peer 26b5292fbabd4fbe9b8146fa205ecb22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0927 22:56:05.428712 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.424859 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.425034 (+ 175us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.425048 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.425132 (+ 84us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.425392 (+ 260us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.425396 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.425405 (+ 9us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.425450 (+ 45us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.425497 (+ 47us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.425503 (+ 6us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.425898 (+ 395us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.425902 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.426400 (+ 498us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.426410 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.427390 (+ 980us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.427397 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.427532 (+ 135us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.427592 (+ 60us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.428329 (+ 737us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.428337 (+ 8us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.428449 (+ 112us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.428504 (+ 55us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":44} I0927 22:56:05.428742 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.424965 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40554 0927 22:56:05.425122 (+ 157us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.425125 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.425141 (+ 16us) server_negotiation.cc:373] Connection header received 0927 22:56:05.425219 (+ 78us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.425223 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.425281 (+ 58us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.425353 (+ 72us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.425605 (+ 252us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.425728 (+ 123us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.426526 (+ 798us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.427137 (+ 611us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.427233 (+ 96us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.427932 (+ 699us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.428063 (+ 131us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.428295 (+ 232us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.428578 (+ 283us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.428606 (+ 28us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":44} I0927 22:56:05.429081 18854 raft_consensus.cc:1058] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Refusing update from remote peer 26b5292fbabd4fbe9b8146fa205ecb22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0927 22:56:05.429894 19111 consensus_queue.cc:962] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00f349d4d9e84c4596252016e4bfc620" member_type: VOTER last_known_addr { host: "127.17.51.1" port: 40273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0927 22:56:05.430342 19119 consensus_queue.cc:962] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54434d11164442ca96a66cda770a149c" member_type: VOTER last_known_addr { host: "127.17.51.3" port: 44871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0927 22:56:05.434854 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.430847 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.431025 (+ 178us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.431105 (+ 80us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.431216 (+ 111us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.431459 (+ 243us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.431462 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.431471 (+ 9us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.431513 (+ 42us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.431560 (+ 47us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.431567 (+ 7us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.431936 (+ 369us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.431940 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.432416 (+ 476us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.432426 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.433245 (+ 819us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.433252 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.433397 (+ 145us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.433461 (+ 64us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.434321 (+ 860us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.434333 (+ 12us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.434493 (+ 160us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.434537 (+ 44us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":74} I0927 22:56:05.435546 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.430924 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40556 0927 22:56:05.431102 (+ 178us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.431107 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.431147 (+ 40us) server_negotiation.cc:373] Connection header received 0927 22:56:05.431313 (+ 166us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.431317 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.431373 (+ 56us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.431441 (+ 68us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.431673 (+ 232us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.431802 (+ 129us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.432581 (+ 779us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.433081 (+ 500us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.433157 (+ 76us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.433829 (+ 672us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.433970 (+ 141us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.434143 (+ 173us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.435290 (+ 1147us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.435332 (+ 42us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":55} I0927 22:56:05.450793 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.444181 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.2:41325 0927 22:56:05.445554 (+ 1373us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.445571 (+ 17us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.445734 (+ 163us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.446463 (+ 729us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.446468 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.446481 (+ 13us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.446867 (+ 386us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.446985 (+ 118us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.446998 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.447471 (+ 473us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.447475 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.447977 (+ 502us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.447987 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.448890 (+ 903us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.448897 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.449046 (+ 149us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.449129 (+ 83us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.450410 (+ 1281us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.450418 (+ 8us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.450532 (+ 114us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.450565 (+ 33us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":141} I0927 22:56:05.450950 19101 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.444615 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40858 0927 22:56:05.444779 (+ 164us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.444783 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.445626 (+ 843us) server_negotiation.cc:373] Connection header received 0927 22:56:05.445917 (+ 291us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.445923 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.446069 (+ 146us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.446167 (+ 98us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.447133 (+ 966us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.447338 (+ 205us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.448135 (+ 797us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.448717 (+ 582us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.448816 (+ 99us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.449762 (+ 946us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.450184 (+ 422us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.450366 (+ 182us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.450659 (+ 293us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.450709 (+ 50us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":46} I0927 22:56:05.465620 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.461697 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.2:41325 0927 22:56:05.461951 (+ 254us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.461966 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.462093 (+ 127us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.462422 (+ 329us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.462426 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.462442 (+ 16us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.462525 (+ 83us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.462585 (+ 60us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.462594 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.462987 (+ 393us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.462991 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.463705 (+ 714us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.463717 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.464562 (+ 845us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.464567 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.464656 (+ 89us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.464708 (+ 52us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.465238 (+ 530us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.465247 (+ 9us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.465357 (+ 110us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.465393 (+ 36us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":93} I0927 22:56:05.465764 19101 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.461697 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40860 0927 22:56:05.461847 (+ 150us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.461850 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.462024 (+ 174us) server_negotiation.cc:373] Connection header received 0927 22:56:05.462198 (+ 174us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.462202 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.462290 (+ 88us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.462366 (+ 76us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.462710 (+ 344us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.462862 (+ 152us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.463871 (+ 1009us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.464377 (+ 506us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.464456 (+ 79us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.464928 (+ 472us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.465029 (+ 101us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.465158 (+ 129us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.465462 (+ 304us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.465495 (+ 33us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":56} I0927 22:56:05.479208 19086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.473785 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:05.474018 (+ 233us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.474036 (+ 18us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.474201 (+ 165us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.475180 (+ 979us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.475189 (+ 9us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.475205 (+ 16us) client_negotiation.cc:175] Negotiated authn=TOKEN 0927 22:56:05.475292 (+ 87us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.475382 (+ 90us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.475395 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.475915 (+ 520us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.475920 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.476646 (+ 726us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.476657 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.477510 (+ 853us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.477517 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.477614 (+ 97us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.477662 (+ 48us) client_negotiation.cc:237] Sending TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.478639 (+ 977us) client_negotiation.cc:254] Received TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.478649 (+ 10us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.478797 (+ 148us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.478849 (+ 52us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":49} I0927 22:56:05.479203 19073 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.473843 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40562 0927 22:56:05.474008 (+ 165us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.474012 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.474152 (+ 140us) server_negotiation.cc:373] Connection header received 0927 22:56:05.474389 (+ 237us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.474396 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.474499 (+ 103us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.474591 (+ 92us) server_negotiation.cc:197] Negotiated authn=TOKEN 0927 22:56:05.475545 (+ 954us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.475753 (+ 208us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.476776 (+ 1023us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.477318 (+ 542us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.477403 (+ 85us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.478169 (+ 766us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0927 22:56:05.478388 (+ 219us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0927 22:56:05.478537 (+ 149us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.478897 (+ 360us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.478945 (+ 48us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":55} I0927 22:56:05.479954 18322 catalog_manager.cc:1666] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40562: table { table_name: "test-table" } I0927 22:56:05.485883 18322 catalog_manager.cc:3806] T 00000000000000000000000000000000 P 133b07c7153a40f0b5e18b033e7982c0: Sending DeleteTablet for 3 replicas of tablet dfb9184aa58e4d0d9661f5db4fb87392 I0927 22:56:05.486001 18322 catalog_manager.cc:2922] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dfb9184aa58e4d0d9661f5db4fb87392 on 00f349d4d9e84c4596252016e4bfc620 (127.17.51.1:40273) (Table deleted at 2018-09-27 22:56:05 UTC) I0927 22:56:05.486165 18322 catalog_manager.cc:2922] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dfb9184aa58e4d0d9661f5db4fb87392 on 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2:41325) (Table deleted at 2018-09-27 22:56:05 UTC) I0927 22:56:05.486301 18322 catalog_manager.cc:2922] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet dfb9184aa58e4d0d9661f5db4fb87392 on 54434d11164442ca96a66cda770a149c (127.17.51.3:44871) (Table deleted at 2018-09-27 22:56:05 UTC) I0927 22:56:05.486912 18574 tablet_service.cc:799] Processing DeleteTablet for tablet dfb9184aa58e4d0d9661f5db4fb87392 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-09-27 22:56:05 UTC) from {username='kudu', principal='kudu/127.17.51.62@KRBTEST.COM'} at 127.0.0.1:32906 I0927 22:56:05.488353 18574 tablet_replica.cc:262] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: stopping tablet replica I0927 22:56:05.488448 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op CompactRowSetsOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488548 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op MinorDeltaCompactionOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488519 18834 tablet_service.cc:799] Processing DeleteTablet for tablet dfb9184aa58e4d0d9661f5db4fb87392 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-09-27 22:56:05 UTC) from {username='kudu', principal='kudu/127.17.51.62@KRBTEST.COM'} at 127.0.0.1:42224 I0927 22:56:05.488602 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op MajorDeltaCompactionOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488654 18834 tablet_replica.cc:262] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: stopping tablet replica I0927 22:56:05.488660 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op UndoDeltaBlockGCOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488720 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op FlushMRSOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488725 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op CompactRowSetsOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488771 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op MinorDeltaCompactionOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488775 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op FlushDeltaMemStoresOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488803 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op MajorDeltaCompactionOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488806 18574 maintenance_manager.cc:235] P 00f349d4d9e84c4596252016e4bfc620: Unregistered op LogGCOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488837 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op UndoDeltaBlockGCOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488878 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op FlushMRSOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488860 18574 raft_consensus.cc:2012] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Raft consensus shutting down. I0927 22:56:05.488960 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op FlushDeltaMemStoresOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488996 18834 maintenance_manager.cc:235] P 54434d11164442ca96a66cda770a149c: Unregistered op LogGCOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.488981 18574 raft_consensus.cc:2039] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620 [term 1 FOLLOWER]: Raft consensus is shut down! I0927 22:56:05.489034 18834 raft_consensus.cc:2012] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Raft consensus shutting down. I0927 22:56:05.489151 18834 raft_consensus.cc:2039] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c [term 1 FOLLOWER]: Raft consensus is shut down! I0927 22:56:05.490161 18705 tablet_service.cc:799] Processing DeleteTablet for tablet dfb9184aa58e4d0d9661f5db4fb87392 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-09-27 22:56:05 UTC) from {username='kudu', principal='kudu/127.17.51.62@KRBTEST.COM'} at 127.0.0.1:40830 I0927 22:56:05.490387 18705 tablet_replica.cc:262] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: stopping tablet replica I0927 22:56:05.490499 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op CompactRowSetsOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.490581 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op MinorDeltaCompactionOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.490638 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op MajorDeltaCompactionOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.490718 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op UndoDeltaBlockGCOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.490787 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op FlushMRSOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.490847 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op FlushDeltaMemStoresOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.490929 18705 maintenance_manager.cc:235] P 26b5292fbabd4fbe9b8146fa205ecb22: Unregistered op LogGCOp(dfb9184aa58e4d0d9661f5db4fb87392) I0927 22:56:05.490958 18834 ts_tablet_manager.cc:1277] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: Deleting tablet data with delete state TABLET_DATA_DELETED I0927 22:56:05.491012 18705 raft_consensus.cc:2012] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 LEADER]: Raft consensus shutting down. I0927 22:56:05.491114 18705 consensus_peers.cc:457] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 -> Peer 54434d11164442ca96a66cda770a149c (127.17.51.3:44871): Closing peer: 54434d11164442ca96a66cda770a149c I0927 22:56:05.491237 18705 consensus_peers.cc:457] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 -> Peer 00f349d4d9e84c4596252016e4bfc620 (127.17.51.1:40273): Closing peer: 00f349d4d9e84c4596252016e4bfc620 I0927 22:56:05.491500 18705 raft_consensus.cc:2039] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22 [term 1 FOLLOWER]: Raft consensus is shut down! I0927 22:56:05.493868 18574 ts_tablet_manager.cc:1277] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: Deleting tablet data with delete state TABLET_DATA_DELETED I0927 22:56:05.497578 18705 ts_tablet_manager.cc:1277] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Deleting tablet data with delete state TABLET_DATA_DELETED I0927 22:56:05.500339 18834 ts_tablet_manager.cc:1290] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0927 22:56:05.500419 18834 log.cc:981] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: Deleting WAL directory at /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-2/wal/wals/dfb9184aa58e4d0d9661f5db4fb87392 I0927 22:56:05.503567 18834 ts_tablet_manager.cc:1310] T dfb9184aa58e4d0d9661f5db4fb87392 P 54434d11164442ca96a66cda770a149c: Deleting consensus metadata I0927 22:56:05.503938 18705 ts_tablet_manager.cc:1290] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0927 22:56:05.504106 18705 log.cc:981] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Deleting WAL directory at /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-1/wal/wals/dfb9184aa58e4d0d9661f5db4fb87392 I0927 22:56:05.504375 17612 external_mini_cluster.cc:1042] Attempting to check leaks for /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver pid 18497 I0927 22:56:05.504681 18705 ts_tablet_manager.cc:1310] T dfb9184aa58e4d0d9661f5db4fb87392 P 26b5292fbabd4fbe9b8146fa205ecb22: Deleting consensus metadata I0927 22:56:05.504719 18309 catalog_manager.cc:2900] TS 54434d11164442ca96a66cda770a149c (127.17.51.3:44871): tablet dfb9184aa58e4d0d9661f5db4fb87392 (table test-table [id=6e2a53728abc48769485469b318e9273]) successfully deleted I0927 22:56:05.505517 18574 ts_tablet_manager.cc:1290] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0927 22:56:05.505615 18574 log.cc:981] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: Deleting WAL directory at /tmp/dist-test-taskFgBoie/test-tmp/security-faults-itest.0.SecurityComponentsFaultsITest.KdcRestartsInTheMiddle.1538088717659548-17612/minicluster-data/ts-0/wal/wals/dfb9184aa58e4d0d9661f5db4fb87392 I0927 22:56:05.506209 18308 catalog_manager.cc:2900] TS 26b5292fbabd4fbe9b8146fa205ecb22 (127.17.51.2:41325): tablet dfb9184aa58e4d0d9661f5db4fb87392 (table test-table [id=6e2a53728abc48769485469b318e9273]) successfully deleted I0927 22:56:05.506269 18574 ts_tablet_manager.cc:1310] T dfb9184aa58e4d0d9661f5db4fb87392 P 00f349d4d9e84c4596252016e4bfc620: Deleting consensus metadata I0927 22:56:05.507436 18306 catalog_manager.cc:2900] TS 00f349d4d9e84c4596252016e4bfc620 (127.17.51.1:40273): tablet dfb9184aa58e4d0d9661f5db4fb87392 (table test-table [id=6e2a53728abc48769485469b318e9273]) successfully deleted Sep 27 22:56:05 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.1@KRBTEST.COM I0927 22:56:05.519702 19132 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.505635 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.1:40273 0927 22:56:05.506823 (+ 1188us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.506840 (+ 17us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.506964 (+ 124us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.507344 (+ 380us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.507347 (+ 3us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.508143 (+ 796us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:56:05.508222 (+ 79us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.508275 (+ 53us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.508283 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.508744 (+ 461us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.508748 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.509195 (+ 447us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.509205 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.510046 (+ 841us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.510051 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.510149 (+ 98us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.510304 (+ 155us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:56:05.510306 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:56:05.513084 (+ 2778us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:56:05.516369 (+ 3285us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:05.516375 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:05.516379 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:05.516886 (+ 507us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:05.517644 (+ 758us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:05.517650 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:05.517653 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:05.517742 (+ 89us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:05.518472 (+ 730us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:56:05.518478 (+ 6us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:56:05.518647 (+ 169us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.518922 (+ 275us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.519275 (+ 353us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":919,"thread_start_us":796,"threads_started":1} I0927 22:56:05.519995 19115 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.505971 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:32936 0927 22:56:05.506165 (+ 194us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.506168 (+ 3us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.506897 (+ 729us) server_negotiation.cc:373] Connection header received 0927 22:56:05.507130 (+ 233us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.507134 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.507214 (+ 80us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.507299 (+ 85us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:56:05.508386 (+ 1087us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.508613 (+ 227us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.509340 (+ 727us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.509903 (+ 563us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.509990 (+ 87us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.513509 (+ 3519us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:56:05.513540 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:56:05.513548 (+ 8us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:56:05.513578 (+ 30us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:56:05.516050 (+ 2472us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:05.517152 (+ 1102us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:05.517158 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:05.517160 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:05.517408 (+ 248us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:05.517981 (+ 573us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:05.517984 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:05.517986 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:05.518220 (+ 234us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:56:05.518456 (+ 236us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.519152 (+ 696us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.519655 (+ 503us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":53} I0927 22:56:05.520619 18634 generic_service.cc:169] Checking for leaks (request via RPC) I0927 22:56:05.839576 17612 external_mini_cluster.cc:1001] Killing /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver with pid 18497 W0927 22:56:05.882410 18285 connection.cc:513] client connection to 127.17.51.1:40273 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:05.882423 18786 connection.cc:513] client connection to 127.17.51.1:40273 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:05.882892 18657 connection.cc:513] client connection to 127.17.51.1:40273 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:05.883352 18438 connection.cc:513] server connection from 127.17.51.1:37893 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:05.883369 18374 connection.cc:513] server connection from 127.17.51.1:35647 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:05.883540 18438 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0927 22:56:05.883574 18374 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0927 22:56:05.883630 18308 connection.cc:513] server connection from 127.17.51.1:34163 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0927 22:56:05.883761 17612 external_mini_cluster.cc:1042] Attempting to check leaks for /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver pid 18648 W0927 22:56:05.883738 18306 connection.cc:513] client connection to 127.17.51.1:40273 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:05.883877 18308 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe Sep 27 22:56:05 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.2@KRBTEST.COM I0927 22:56:05.896878 19132 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.884282 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.2:41325 0927 22:56:05.884584 (+ 302us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:05.884600 (+ 16us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:05.884697 (+ 97us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:05.885044 (+ 347us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:05.885050 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:05.886110 (+ 1060us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:56:05.886189 (+ 79us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.886264 (+ 75us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.886276 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.886912 (+ 636us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.886917 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.887408 (+ 491us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:05.887418 (+ 10us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.888554 (+ 1136us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.888559 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:05.888657 (+ 98us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.888780 (+ 123us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:56:05.888782 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:56:05.891218 (+ 2436us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:56:05.894327 (+ 3109us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:05.894331 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:05.894333 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:05.894601 (+ 268us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:05.895323 (+ 722us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:05.895326 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:05.895328 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:05.895373 (+ 45us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:05.896023 (+ 650us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:56:05.896027 (+ 4us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:56:05.896151 (+ 124us) client_negotiation.cc:695] Sending connection context 0927 22:56:05.896334 (+ 183us) client_negotiation.cc:225] Negotiation successful 0927 22:56:05.896619 (+ 285us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":104} I0927 22:56:05.897197 19101 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:05.884406 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40866 0927 22:56:05.884644 (+ 238us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:05.884649 (+ 5us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:05.884667 (+ 18us) server_negotiation.cc:373] Connection header received 0927 22:56:05.884794 (+ 127us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:05.884798 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:05.884865 (+ 67us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:05.884961 (+ 96us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:56:05.886520 (+ 1559us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.886782 (+ 262us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.887547 (+ 765us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:05.888364 (+ 817us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:05.888485 (+ 121us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:05.891474 (+ 2989us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:56:05.891515 (+ 41us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:56:05.891523 (+ 8us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:56:05.891559 (+ 36us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:56:05.894100 (+ 2541us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:05.894829 (+ 729us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:05.894836 (+ 7us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:05.894839 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:05.895127 (+ 288us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:05.895563 (+ 436us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:05.895566 (+ 3us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:05.895568 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:05.895809 (+ 241us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:56:05.896114 (+ 305us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:05.896450 (+ 336us) server_negotiation.cc:264] Negotiation successful 0927 22:56:05.896906 (+ 456us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":97} I0927 22:56:05.897739 18765 generic_service.cc:169] Checking for leaks (request via RPC) I0927 22:56:06.183522 17612 external_mini_cluster.cc:1001] Killing /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver with pid 18648 W0927 22:56:06.221835 18285 connection.cc:513] client connection to 127.17.51.2:41325 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.221912 18788 connection.cc:513] client connection to 127.17.51.2:41325 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.221940 18437 connection.cc:513] server connection from 127.17.51.2:45075 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.222147 18437 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0927 22:56:06.222281 18308 connection.cc:513] client connection to 127.17.51.2:41325 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0927 22:56:06.222331 17612 external_mini_cluster.cc:1042] Attempting to check leaks for /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver pid 18777 W0927 22:56:06.222319 18372 connection.cc:513] server connection from 127.17.51.2:58565 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.222493 18372 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0927 22:56:06.222627 18309 connection.cc:513] server connection from 127.17.51.2:44281 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.222659 18786 connection.cc:513] server connection from 127.17.51.2:52109 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.222770 18309 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe Sep 27 22:56:06 dist-test-slave-dist-test-slave-xppb krb5kdc[19076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1538088965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.51.3@KRBTEST.COM I0927 22:56:06.235427 19132 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.222919 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.3:44871 0927 22:56:06.223160 (+ 241us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:06.223175 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:06.223277 (+ 102us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:06.223751 (+ 474us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:06.223755 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:06.224523 (+ 768us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:56:06.224586 (+ 63us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.224645 (+ 59us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:06.224653 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.225037 (+ 384us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.225040 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.225459 (+ 419us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:06.225468 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.226253 (+ 785us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.226258 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.226360 (+ 102us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:06.226480 (+ 120us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:56:06.226482 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:56:06.229183 (+ 2701us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:56:06.232255 (+ 3072us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:06.232260 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:06.232262 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:06.232605 (+ 343us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:06.233477 (+ 872us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:06.233482 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:06.233485 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:06.233560 (+ 75us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:06.234622 (+ 1062us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:56:06.234626 (+ 4us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:56:06.234740 (+ 114us) client_negotiation.cc:695] Sending connection context 0927 22:56:06.234901 (+ 161us) client_negotiation.cc:225] Negotiation successful 0927 22:56:06.235158 (+ 257us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":72} I0927 22:56:06.236124 19135 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.222988 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42260 0927 22:56:06.223477 (+ 489us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:06.223481 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:06.223498 (+ 17us) server_negotiation.cc:373] Connection header received 0927 22:56:06.223548 (+ 50us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:06.223552 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:06.223628 (+ 76us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:06.223722 (+ 94us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:56:06.224760 (+ 1038us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.224936 (+ 176us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.225567 (+ 631us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.226084 (+ 517us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.226166 (+ 82us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:06.229469 (+ 3303us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:56:06.229511 (+ 42us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:56:06.229518 (+ 7us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:56:06.229555 (+ 37us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:56:06.231975 (+ 2420us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:06.232851 (+ 876us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:06.232857 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:06.232859 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:06.233198 (+ 339us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:06.233759 (+ 561us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:06.233764 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:06.233767 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:06.234106 (+ 339us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:56:06.234486 (+ 380us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:06.235113 (+ 627us) server_negotiation.cc:264] Negotiation successful 0927 22:56:06.235732 (+ 619us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":192,"threads_started":1} I0927 22:56:06.236778 18894 generic_service.cc:169] Checking for leaks (request via RPC) I0927 22:56:06.514240 17612 external_mini_cluster.cc:1001] Killing /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-tserver with pid 18777 W0927 22:56:06.550020 18309 connection.cc:513] client connection to 127.17.51.3:44871 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.550076 18437 connection.cc:513] server connection from 127.17.51.3:34323 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.550082 18285 connection.cc:513] client connection to 127.17.51.3:44871 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.550293 18437 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0927 22:56:06.550331 17612 external_mini_cluster.cc:1042] Attempting to check leaks for /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master pid 18297 W0927 22:56:06.550469 18374 connection.cc:513] server connection from 127.17.51.3:54491 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.550627 18374 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0927 22:56:06.550598 18306 connection.cc:513] server connection from 127.17.51.3:43083 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0927 22:56:06.562530 19132 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.550906 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:06.551203 (+ 297us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:06.551222 (+ 19us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:06.551347 (+ 125us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:06.551750 (+ 403us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:06.551754 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:06.552633 (+ 879us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:56:06.552800 (+ 167us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.552860 (+ 60us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:06.552872 (+ 12us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.553406 (+ 534us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.553410 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.553864 (+ 454us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:06.553873 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.554937 (+ 1064us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.554944 (+ 7us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.555105 (+ 161us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:06.555263 (+ 158us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:56:06.555265 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:56:06.556307 (+ 1042us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:56:06.559699 (+ 3392us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:06.559703 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:06.559706 (+ 3us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:06.559972 (+ 266us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:06.560769 (+ 797us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:06.560773 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:06.560775 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:06.560824 (+ 49us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:06.561487 (+ 663us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:56:06.561492 (+ 5us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:56:06.561630 (+ 138us) client_negotiation.cc:695] Sending connection context 0927 22:56:06.561816 (+ 186us) client_negotiation.cc:225] Negotiation successful 0927 22:56:06.562159 (+ 343us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":93} I0927 22:56:06.562904 19137 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.550996 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:40570 0927 22:56:06.551431 (+ 435us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:06.551435 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:06.551454 (+ 19us) server_negotiation.cc:373] Connection header received 0927 22:56:06.551513 (+ 59us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:06.551517 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:06.551604 (+ 87us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:06.551696 (+ 92us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:56:06.553014 (+ 1318us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.553256 (+ 242us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.553993 (+ 737us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.554769 (+ 776us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.554899 (+ 130us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:06.556609 (+ 1710us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:56:06.556649 (+ 40us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:56:06.556655 (+ 6us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:56:06.556696 (+ 41us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:56:06.559456 (+ 2760us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:06.560205 (+ 749us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:06.560210 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:06.560211 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:06.560442 (+ 231us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:06.561027 (+ 585us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:06.561031 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:06.561032 (+ 1us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:06.561237 (+ 205us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:56:06.561505 (+ 268us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:06.561998 (+ 493us) server_negotiation.cc:264] Negotiation successful 0927 22:56:06.562668 (+ 670us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":307,"thread_start_us":171,"threads_started":1} I0927 22:56:06.563370 18354 generic_service.cc:169] Checking for leaks (request via RPC) I0927 22:56:06.636220 19139 raft_consensus.cc:436] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 133b07c7153a40f0b5e18b033e7982c0) I0927 22:56:06.636317 19139 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Snoozing failure detection for 0.109s (starting election) I0927 22:56:06.636410 19139 raft_consensus.cc:458] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:56:06.636790 19140 raft_consensus.cc:436] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 133b07c7153a40f0b5e18b033e7982c0) I0927 22:56:06.636879 19140 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.096s (starting election) I0927 22:56:06.636960 19140 raft_consensus.cc:458] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:56:06.637501 19139 leader_election.cc:231] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 133b07c7153a40f0b5e18b033e7982c0 I0927 22:56:06.637552 19140 leader_election.cc:231] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 133b07c7153a40f0b5e18b033e7982c0 I0927 22:56:06.637725 19140 leader_election.cc:231] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 93ce3bfa2f2a4d00abf078a11b3b7fec I0927 22:56:06.637768 19139 leader_election.cc:231] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 9fa6f72caa24460e896f673a01c0fcf2 I0927 22:56:06.650161 19143 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.638160 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:56:06.644717 (+ 6557us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:06.644731 (+ 14us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:06.644808 (+ 77us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:06.646249 (+ 1441us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:06.646255 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:06.646287 (+ 32us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:56:06.646401 (+ 114us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.646486 (+ 85us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:06.646497 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.646985 (+ 488us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.646990 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.648161 (+ 1171us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:06.648172 (+ 11us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.649385 (+ 1213us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.649391 (+ 6us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.649587 (+ 196us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:06.649642 (+ 55us) client_negotiation.cc:695] Sending connection context 0927 22:56:06.649809 (+ 167us) client_negotiation.cc:225] Negotiation successful 0927 22:56:06.649854 (+ 45us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":6457,"mutex_wait_us":3,"thread_start_us":6395,"threads_started":1} I0927 22:56:06.650249 19146 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.645342 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42666 0927 22:56:06.645996 (+ 654us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:06.646000 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:06.646014 (+ 14us) server_negotiation.cc:373] Connection header received 0927 22:56:06.646069 (+ 55us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:06.646073 (+ 4us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:06.646134 (+ 61us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:06.646209 (+ 75us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:56:06.646623 (+ 414us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.646856 (+ 233us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.648283 (+ 1427us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.649200 (+ 917us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.649295 (+ 95us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:06.649446 (+ 151us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:06.649958 (+ 512us) server_negotiation.cc:264] Negotiation successful 0927 22:56:06.650001 (+ 43us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":547,"thread_start_us":442,"threads_started":1} I0927 22:56:06.650290 19144 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.638027 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:56:06.644749 (+ 6722us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:06.644764 (+ 15us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:06.644894 (+ 130us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:06.645592 (+ 698us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:06.645596 (+ 4us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:06.645620 (+ 24us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:56:06.645685 (+ 65us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.645897 (+ 212us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:06.645911 (+ 14us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.646420 (+ 509us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.646425 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.647660 (+ 1235us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:06.647673 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.649570 (+ 1897us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.649589 (+ 19us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.649729 (+ 140us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:06.649779 (+ 50us) client_negotiation.cc:695] Sending connection context 0927 22:56:06.649956 (+ 177us) client_negotiation.cc:225] Negotiation successful 0927 22:56:06.649996 (+ 40us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":6571,"mutex_wait_us":1,"thread_start_us":7088,"threads_started":1} I0927 22:56:06.650854 19145 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.644717 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51368 0927 22:56:06.645262 (+ 545us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:06.645268 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:06.645290 (+ 22us) server_negotiation.cc:373] Connection header received 0927 22:56:06.645366 (+ 76us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:06.645371 (+ 5us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:06.645458 (+ 87us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:06.645544 (+ 86us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0927 22:56:06.646041 (+ 497us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.646284 (+ 243us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.647813 (+ 1529us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.649208 (+ 1395us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.649295 (+ 87us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:06.649503 (+ 208us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:06.650627 (+ 1124us) server_negotiation.cc:264] Negotiation successful 0927 22:56:06.650669 (+ 42us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":423,"thread_start_us":333,"threads_started":1} I0927 22:56:06.651129 18464 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "9fa6f72caa24460e896f673a01c0fcf2" is_pre_election: true I0927 22:56:06.651285 18464 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.123s (vote granted) I0927 22:56:06.651340 18464 raft_consensus.cc:2215] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93ce3bfa2f2a4d00abf078a11b3b7fec in term 1. I0927 22:56:06.651373 18398 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fa6f72caa24460e896f673a01c0fcf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" is_pre_election: true I0927 22:56:06.651537 18398 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Snoozing failure detection for 0.082s (vote granted) I0927 22:56:06.651590 18398 raft_consensus.cc:2215] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fa6f72caa24460e896f673a01c0fcf2 in term 1. I0927 22:56:06.651902 18373 leader_election.cc:363] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 2 pre-election: Vote granted by peer 9fa6f72caa24460e896f673a01c0fcf2 I0927 22:56:06.651976 18373 leader_election.cc:258] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0927 22:56:06.652292 19139 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Snoozing failure detection for 0.096s (election complete) I0927 22:56:06.652365 19139 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 1 FOLLOWER]: Leader pre-election won for term 2 I0927 22:56:06.652411 19139 raft_consensus.cc:436] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 1 FOLLOWER]: Starting leader election (detected failure of leader 133b07c7153a40f0b5e18b033e7982c0) I0927 22:56:06.652462 19139 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Snoozing failure detection for 0.097s (starting election) I0927 22:56:06.652526 19139 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 1 FOLLOWER]: Advancing to term 2 I0927 22:56:06.652626 18436 leader_election.cc:363] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 93ce3bfa2f2a4d00abf078a11b3b7fec I0927 22:56:06.652683 18436 leader_election.cc:258] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0927 22:56:06.652892 19140 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.083s (election complete) I0927 22:56:06.652983 19140 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0927 22:56:06.653048 19140 raft_consensus.cc:436] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 133b07c7153a40f0b5e18b033e7982c0) I0927 22:56:06.653132 19140 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.081s (starting election) I0927 22:56:06.653199 19140 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 1 FOLLOWER]: Advancing to term 2 I0927 22:56:06.654724 19139 raft_consensus.cc:458] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:56:06.655326 19139 leader_election.cc:231] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 2 election: Requesting vote from peer 133b07c7153a40f0b5e18b033e7982c0 I0927 22:56:06.655339 19140 raft_consensus.cc:458] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:56:06.655516 19139 leader_election.cc:231] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 2 election: Requesting vote from peer 9fa6f72caa24460e896f673a01c0fcf2 I0927 22:56:06.656229 19140 leader_election.cc:231] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 2 election: Requesting vote from peer 133b07c7153a40f0b5e18b033e7982c0 I0927 22:56:06.656190 18464 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "9fa6f72caa24460e896f673a01c0fcf2" I0927 22:56:06.656412 18464 raft_consensus.cc:2145] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 93ce3bfa2f2a4d00abf078a11b3b7fec in current term 2: Already voted for candidate 9fa6f72caa24460e896f673a01c0fcf2 in this term. I0927 22:56:06.656612 19140 leader_election.cc:231] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 2 election: Requesting vote from peer 93ce3bfa2f2a4d00abf078a11b3b7fec I0927 22:56:06.657001 18373 leader_election.cc:377] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 2 election: Vote denied by peer 9fa6f72caa24460e896f673a01c0fcf2. Message: Invalid argument: T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 93ce3bfa2f2a4d00abf078a11b3b7fec in current term 2: Already voted for candidate 9fa6f72caa24460e896f673a01c0fcf2 in this term. I0927 22:56:06.657441 18398 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fa6f72caa24460e896f673a01c0fcf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" I0927 22:56:06.657629 18398 raft_consensus.cc:2145] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9fa6f72caa24460e896f673a01c0fcf2 in current term 2: Already voted for candidate 93ce3bfa2f2a4d00abf078a11b3b7fec in this term. I0927 22:56:06.658114 18436 leader_election.cc:377] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 2 election: Vote denied by peer 93ce3bfa2f2a4d00abf078a11b3b7fec. Message: Invalid argument: T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9fa6f72caa24460e896f673a01c0fcf2 in current term 2: Already voted for candidate 93ce3bfa2f2a4d00abf078a11b3b7fec in this term. I0927 22:56:06.735291 19140 raft_consensus.cc:436] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0927 22:56:06.735399 19140 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.146s (starting election) I0927 22:56:06.735491 19140 raft_consensus.cc:458] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:56:06.736109 19140 leader_election.cc:231] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 133b07c7153a40f0b5e18b033e7982c0 I0927 22:56:06.736315 19140 leader_election.cc:231] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 93ce3bfa2f2a4d00abf078a11b3b7fec I0927 22:56:06.737340 18398 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fa6f72caa24460e896f673a01c0fcf2" candidate_term: 3 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" is_pre_election: true I0927 22:56:06.737543 18398 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Snoozing failure detection for 0.111s (vote granted) W0927 22:56:06.737599 18437 leader_election.cc:287] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 133b07c7153a40f0b5e18b033e7982c0: Timed out: connection negotiation to 127.17.51.62:11080 for RPC RequestConsensusVote timed out after 0.096s (ON_OUTBOUND_QUEUE) I0927 22:56:06.737615 18398 raft_consensus.cc:2215] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fa6f72caa24460e896f673a01c0fcf2 in term 2. W0927 22:56:06.737773 18437 leader_election.cc:287] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 133b07c7153a40f0b5e18b033e7982c0: Timed out: connection negotiation to 127.17.51.62:11080 for RPC RequestConsensusVote timed out after 0.081s (ON_OUTBOUND_QUEUE) I0927 22:56:06.737820 18437 leader_election.cc:258] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I0927 22:56:06.738240 18436 leader_election.cc:363] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 3 pre-election: Vote granted by peer 93ce3bfa2f2a4d00abf078a11b3b7fec I0927 22:56:06.738299 18436 leader_election.cc:258] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0927 22:56:06.738555 19140 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.130s (election complete) I0927 22:56:06.738612 19140 raft_consensus.cc:2455] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0927 22:56:06.738693 19140 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.207s (election complete) I0927 22:56:06.738745 19140 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0927 22:56:06.738788 19140 raft_consensus.cc:436] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0927 22:56:06.738826 19140 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.227s (starting election) I0927 22:56:06.738868 19140 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 2 FOLLOWER]: Advancing to term 3 I0927 22:56:06.741216 19140 raft_consensus.cc:458] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:56:06.741737 19140 leader_election.cc:231] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 3 election: Requesting vote from peer 133b07c7153a40f0b5e18b033e7982c0 I0927 22:56:06.741881 19140 leader_election.cc:231] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 3 election: Requesting vote from peer 93ce3bfa2f2a4d00abf078a11b3b7fec I0927 22:56:06.742703 18398 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9fa6f72caa24460e896f673a01c0fcf2" candidate_term: 3 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" I0927 22:56:06.742851 18398 raft_consensus.cc:2752] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 2 FOLLOWER]: Advancing to term 3 I0927 22:56:06.742925 18398 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Snoozing failure detection for 0.083s (vote granted) I0927 22:56:06.744513 18398 raft_consensus.cc:2215] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fa6f72caa24460e896f673a01c0fcf2 in term 3. I0927 22:56:06.745040 18436 leader_election.cc:363] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 3 election: Vote granted by peer 93ce3bfa2f2a4d00abf078a11b3b7fec I0927 22:56:06.745111 18436 leader_election.cc:258] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0927 22:56:06.745311 19140 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Snoozing failure detection for 0.094s (election complete) I0927 22:56:06.745388 19140 raft_consensus.cc:2507] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 3 FOLLOWER]: Leader election won for term 3 I0927 22:56:06.745530 19140 raft_consensus.cc:572] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 3 LEADER]: Becoming Leader. State: Replica: 9fa6f72caa24460e896f673a01c0fcf2, State: Running, Role: LEADER I0927 22:56:06.745941 19140 consensus_queue.cc:203] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 16, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } W0927 22:56:06.747014 18372 leader_election.cc:287] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 133b07c7153a40f0b5e18b033e7982c0: Timed out: connection negotiation to 127.17.51.62:11080 for RPC RequestConsensusVote timed out after 0.109s (ON_OUTBOUND_QUEUE) I0927 22:56:06.747664 19147 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa6f72caa24460e896f673a01c0fcf2. Latest consensus state: current_term: 3 leader_uuid: "9fa6f72caa24460e896f673a01c0fcf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:56:06.747905 19147 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: This master's current role is: LEADER I0927 22:56:06.748370 19149 catalog_manager.cc:972] Loading table and tablet metadata into memory... W0927 22:56:06.753543 18372 leader_election.cc:287] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 133b07c7153a40f0b5e18b033e7982c0: Timed out: connection negotiation to 127.17.51.62:11080 for RPC RequestConsensusVote timed out after 0.097s (ON_OUTBOUND_QUEUE) I0927 22:56:06.753636 18372 leader_election.cc:258] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I0927 22:56:06.753901 19139 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Snoozing failure detection for 0.086s (election complete) I0927 22:56:06.753965 19139 raft_consensus.cc:2455] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0927 22:56:06.754606 19149 catalog_manager.cc:983] Initializing Kudu internal certificate authority... I0927 22:56:06.756228 19149 catalog_manager.cc:992] Loading token signing keys... I0927 22:56:06.757491 19149 catalog_manager.cc:3882] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2: Loaded TSK: 0 I0927 22:56:06.762709 18398 raft_consensus.cc:1058] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 3 FOLLOWER]: Refusing update from remote peer 9fa6f72caa24460e896f673a01c0fcf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 3 index: 18. (index mismatch) I0927 22:56:06.763461 19147 consensus_queue.cc:962] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s I0927 22:56:06.769243 19139 sys_catalog.cc:335] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa6f72caa24460e896f673a01c0fcf2. Latest consensus state: current_term: 3 leader_uuid: "9fa6f72caa24460e896f673a01c0fcf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:56:06.769454 19139 sys_catalog.cc:338] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: This master's current role is: FOLLOWER I0927 22:56:06.771606 19147 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9fa6f72caa24460e896f673a01c0fcf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:56:06.771826 19147 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: This master's current role is: LEADER I0927 22:56:06.772325 19139 sys_catalog.cc:335] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9fa6f72caa24460e896f673a01c0fcf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:56:06.772508 19139 sys_catalog.cc:338] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [sys.catalog]: This master's current role is: FOLLOWER W0927 22:56:06.883713 18437 leader_election.cc:287] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 133b07c7153a40f0b5e18b033e7982c0: Timed out: connection negotiation to 127.17.51.62:11080 for RPC RequestConsensusVote timed out after 0.146s (ON_OUTBOUND_QUEUE) I0927 22:56:06.884407 17612 external_mini_cluster.cc:1001] Killing /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master with pid 18297 I0927 22:56:06.884690 18398 raft_consensus.cc:1026] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 3 FOLLOWER]: Rejecting Update request from peer 133b07c7153a40f0b5e18b033e7982c0 for earlier term 1. Current term is 3. Ops: [] W0927 22:56:06.931038 18285 connection.cc:513] client connection to 127.17.51.62:11080 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.931134 18436 connection.cc:513] server connection from 127.0.0.1:42496 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.931254 18436 connection.cc:149] Shutting down server connection from 127.0.0.1:42496 with pending inbound data (64/190 bytes received, last active 0 ns ago, status=Network error: failed to read from TLS socket: Connection reset by peer (error 104)) W0927 22:56:06.931413 18374 connection.cc:513] server connection from 127.0.0.1:51188 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:06.931464 19142 negotiation.cc:313] Failed RPC negotiation. Trace: 0927 22:56:06.638029 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:06.644031 (+ 6002us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:06.644049 (+ 18us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:06.644214 (+ 165us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:06.931169 (+286955us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: BlockingRecv error: recv error: Connection reset by peer (error 104) Metrics: {"client-negotiator.queue_time_us":5839,"thread_start_us":6226,"threads_started":1} W0927 22:56:06.931682 18374 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0927 22:56:06.931973 19141 negotiation.cc:313] Failed RPC negotiation. Trace: 0927 22:56:06.640489 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:06.645335 (+ 4846us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:06.645346 (+ 11us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:06.645417 (+ 71us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:06.884927 (+239510us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:06.884933 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:06.884989 (+ 56us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0927 22:56:06.885135 (+ 146us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.885231 (+ 96us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:06.885244 (+ 13us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.931737 (+ 46493us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: BlockingRecv error: recv error: Connection reset by peer (error 104) Metrics: {"client-negotiator.queue_time_us":4751,"thread_start_us":4038,"threads_started":1} I0927 22:56:06.931977 17612 external_mini_cluster.cc:1042] Attempting to check leaks for /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master pid 18362 W0927 22:56:06.932210 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: BlockingRecv error: recv error: Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. W0927 22:56:06.932344 18437 leader_election.cc:287] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 133b07c7153a40f0b5e18b033e7982c0: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: BlockingRecv error: recv error: Connection reset by peer (error 104) I0927 22:56:06.933833 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.933375 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:06.933590 (+ 215us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:06.933683 (+ 93us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":81} W0927 22:56:06.933985 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I0927 22:56:06.945017 19132 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.932607 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:56:06.932868 (+ 261us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:06.932886 (+ 18us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:06.933002 (+ 116us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:06.933485 (+ 483us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:06.933490 (+ 5us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:06.934462 (+ 972us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:56:06.934530 (+ 68us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.934607 (+ 77us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:06.934616 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.935011 (+ 395us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.935014 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.935455 (+ 441us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:06.935464 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.936676 (+ 1212us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.936681 (+ 5us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:06.936806 (+ 125us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:06.936955 (+ 149us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:56:06.936960 (+ 5us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:56:06.938152 (+ 1192us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:56:06.942261 (+ 4109us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:06.942267 (+ 6us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:06.942271 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:06.942552 (+ 281us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:06.943383 (+ 831us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:06.943386 (+ 3us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:06.943388 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:06.943430 (+ 42us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:06.944154 (+ 724us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:56:06.944157 (+ 3us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:56:06.944266 (+ 109us) client_negotiation.cc:695] Sending connection context 0927 22:56:06.944414 (+ 148us) client_negotiation.cc:225] Negotiation successful 0927 22:56:06.944746 (+ 332us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":92} I0927 22:56:06.945533 19145 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.932709 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:51374 0927 22:56:06.932900 (+ 191us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:06.932904 (+ 4us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:06.932924 (+ 20us) server_negotiation.cc:373] Connection header received 0927 22:56:06.933198 (+ 274us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:06.933205 (+ 7us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:06.933313 (+ 108us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:06.933426 (+ 113us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:56:06.934740 (+ 1314us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.934901 (+ 161us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.935626 (+ 725us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:06.936507 (+ 881us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:06.936617 (+ 110us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:06.938443 (+ 1826us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:56:06.938475 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:56:06.938480 (+ 5us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:56:06.938513 (+ 33us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:56:06.941954 (+ 3441us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:06.942797 (+ 843us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:06.942803 (+ 6us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:06.942806 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:06.943170 (+ 364us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:06.943672 (+ 502us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:06.943677 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:06.943680 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:06.943919 (+ 239us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:56:06.944230 (+ 311us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:06.944651 (+ 421us) server_negotiation.cc:264] Negotiation successful 0927 22:56:06.945247 (+ 596us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":49} I0927 22:56:06.946100 18419 generic_service.cc:169] Checking for leaks (request via RPC) I0927 22:56:06.962570 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.962104 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:06.962316 (+ 212us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:06.962388 (+ 72us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":78} W0927 22:56:06.962755 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. I0927 22:56:06.984431 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:06.983909 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:06.984173 (+ 264us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:06.984250 (+ 77us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":84} W0927 22:56:06.984612 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. I0927 22:56:07.013881 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.013362 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.013613 (+ 251us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.013686 (+ 73us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":94} W0927 22:56:07.014096 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. I0927 22:56:07.041389 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.040885 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.041144 (+ 259us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.041217 (+ 73us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":90} W0927 22:56:07.041563 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. I0927 22:56:07.069293 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.068747 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.068989 (+ 242us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.069091 (+ 102us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":96} W0927 22:56:07.069478 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. I0927 22:56:07.098829 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.098327 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.098567 (+ 240us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.098650 (+ 83us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":94} W0927 22:56:07.098994 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. I0927 22:56:07.125524 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.124983 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.125250 (+ 267us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.125322 (+ 72us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":114} W0927 22:56:07.125708 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. I0927 22:56:07.149241 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.148725 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.148963 (+ 238us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.149036 (+ 73us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":87} W0927 22:56:07.149410 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. I0927 22:56:07.174933 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.174435 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.174687 (+ 252us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.174759 (+ 72us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":95} W0927 22:56:07.175160 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times. I0927 22:56:07.205651 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.205149 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.205392 (+ 243us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.205465 (+ 73us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":100} W0927 22:56:07.205818 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 12 times. I0927 22:56:07.235402 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.234851 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.235127 (+ 276us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.235201 (+ 74us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":97} W0927 22:56:07.235585 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 13 times. I0927 22:56:07.257565 19139 raft_consensus.cc:436] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9fa6f72caa24460e896f673a01c0fcf2) I0927 22:56:07.257972 19139 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Snoozing failure detection for 0.080s (starting election) I0927 22:56:07.258210 19139 raft_consensus.cc:458] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } I0927 22:56:07.259183 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.258624 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.258866 (+ 242us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.258954 (+ 88us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":82} I0927 22:56:07.259238 19139 leader_election.cc:231] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 133b07c7153a40f0b5e18b033e7982c0 W0927 22:56:07.259354 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 14 times. I0927 22:56:07.259450 19139 leader_election.cc:231] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 9fa6f72caa24460e896f673a01c0fcf2 I0927 22:56:07.276249 18464 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "9fa6f72caa24460e896f673a01c0fcf2" is_pre_election: true I0927 22:56:07.276522 18464 raft_consensus.cc:2176] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 93ce3bfa2f2a4d00abf078a11b3b7fec for term 4 because replica is either leader or believes a valid leader to be alive. I0927 22:56:07.277027 19142 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.259606 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.259827 (+ 221us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.259895 (+ 68us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":87} I0927 22:56:07.277506 18373 leader_election.cc:377] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 4 pre-election: Vote denied by peer 9fa6f72caa24460e896f673a01c0fcf2. Message: Invalid argument: T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 93ce3bfa2f2a4d00abf078a11b3b7fec for term 4 because replica is either leader or believes a valid leader to be alive. W0927 22:56:07.277876 18372 leader_election.cc:287] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 133b07c7153a40f0b5e18b033e7982c0: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) I0927 22:56:07.278219 18372 leader_election.cc:258] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. I0927 22:56:07.278949 19139 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Snoozing failure detection for 0.109s (election complete) I0927 22:56:07.279129 19139 raft_consensus.cc:2455] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0927 22:56:07.279415 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.278792 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.279112 (+ 320us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.279199 (+ 87us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":137} W0927 22:56:07.279522 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 15 times. I0927 22:56:07.299796 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.299318 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.299547 (+ 229us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.299621 (+ 74us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":84} W0927 22:56:07.299959 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 16 times. I0927 22:56:07.329532 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.328985 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.329229 (+ 244us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.329316 (+ 87us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":97} W0927 22:56:07.329717 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 17 times. I0927 22:56:07.353013 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.352510 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.352721 (+ 211us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.352807 (+ 86us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":67} W0927 22:56:07.353224 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 18 times. I0927 22:56:07.379705 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.379184 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.379405 (+ 221us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.379494 (+ 89us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":70} W0927 22:56:07.379907 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 19 times. I0927 22:56:07.408043 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.407527 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.407746 (+ 219us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.407843 (+ 97us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":72} W0927 22:56:07.408249 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 20 times. I0927 22:56:07.433616 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.433085 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.433308 (+ 223us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.433395 (+ 87us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":75} W0927 22:56:07.433799 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 21 times. I0927 22:56:07.451983 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.451505 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.451714 (+ 209us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.451796 (+ 82us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":71} W0927 22:56:07.452172 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 22 times. I0927 22:56:07.479600 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.479049 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.479303 (+ 254us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.479389 (+ 86us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":71} W0927 22:56:07.479775 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 23 times. I0927 22:56:07.506152 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.505617 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.505829 (+ 212us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.505912 (+ 83us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":71} W0927 22:56:07.506326 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 24 times. I0927 22:56:07.530663 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.530151 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.530366 (+ 215us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.530450 (+ 84us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":72} W0927 22:56:07.530859 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 25 times. I0927 22:56:07.548907 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.548401 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.548628 (+ 227us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.548702 (+ 74us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":82} W0927 22:56:07.549123 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 26 times. I0927 22:56:07.578685 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.578171 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.578401 (+ 230us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.578479 (+ 78us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":94} W0927 22:56:07.578871 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 27 times. ================================================================= ==18362==ERROR: LeakSanitizer: detected memory leaks Direct leak of 8 byte(s) in 1 object(s) allocated from: #0 0x4e4b25 in calloc /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:74 #1 0x7f06d7c7d9f2 (/usr/lib/x86_64-linux-gnu/libkrb5.so.3+0x4e9f2) SUMMARY: AddressSanitizer: 8 byte(s) leaked in 1 allocation(s). I0927 22:56:07.608387 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.607890 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.608139 (+ 249us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.608216 (+ 77us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":84} W0927 22:56:07.608556 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 28 times. I0927 22:56:07.628151 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.627581 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.627838 (+ 257us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.627916 (+ 78us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":116} W0927 22:56:07.628348 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 29 times. I0927 22:56:07.655942 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.655454 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.655679 (+ 225us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.655759 (+ 80us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":79} W0927 22:56:07.656149 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 30 times. I0927 22:56:07.684180 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.683665 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.683886 (+ 221us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.683964 (+ 78us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":88} W0927 22:56:07.684366 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 31 times. I0927 22:56:07.718781 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.718273 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.718504 (+ 231us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.718594 (+ 90us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":89} W0927 22:56:07.718958 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 32 times. I0927 22:56:07.748332 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.747747 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.747974 (+ 227us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.748095 (+ 121us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":43} W0927 22:56:07.748545 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 33 times. I0927 22:56:07.772104 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.771474 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.771726 (+ 252us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.771824 (+ 98us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":89} W0927 22:56:07.772317 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 34 times. I0927 22:56:07.796641 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.796092 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.796308 (+ 216us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.796386 (+ 78us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":78} W0927 22:56:07.796821 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 35 times. I0927 22:56:07.817178 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.816585 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.816898 (+ 313us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.816974 (+ 76us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":182} W0927 22:56:07.817342 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 36 times. I0927 22:56:07.841768 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.841244 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.841476 (+ 232us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.841555 (+ 79us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":97} W0927 22:56:07.841958 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 37 times. I0927 22:56:07.871567 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.871043 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.871310 (+ 267us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.871384 (+ 74us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":89} W0927 22:56:07.871750 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 38 times. I0927 22:56:07.898387 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.897867 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.898104 (+ 237us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.898197 (+ 93us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":67} W0927 22:56:07.898578 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 39 times. I0927 22:56:07.910490 19139 raft_consensus.cc:436] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9fa6f72caa24460e896f673a01c0fcf2) ================================================================= ==18362==ERROR: LeakSanitizer: detected memory leaks Direct leak of 8 byte(s) in 1 object(s) allocated from: I0927 22:56:07.910876 19139 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Snoozing failure detection for 0.077s (starting election) #0 0x4e4b25 in calloc /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-4.0.0.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:74 #1 0x7f06d7c7d9f2 (/usr/lib/x86_64-linux-gnu/libkrb5.so.3+0x4e9f2) I0927 22:56:07.911365 19139 raft_consensus.cc:458] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } SUMMARY: AddressSanitizer: 8 byte(s) leaked in 1 allocation(s). I0927 22:56:07.912019 19139 leader_election.cc:231] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 133b07c7153a40f0b5e18b033e7982c0 I0927 22:56:07.912259 19139 leader_election.cc:231] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 9fa6f72caa24460e896f673a01c0fcf2 I0927 22:56:07.913261 19159 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.912446 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.912908 (+ 462us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.913003 (+ 95us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":182,"threads_started":1} I0927 22:56:07.913327 18464 tablet_service.cc:994] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "9fa6f72caa24460e896f673a01c0fcf2" is_pre_election: true W0927 22:56:07.913475 18372 leader_election.cc:287] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 133b07c7153a40f0b5e18b033e7982c0: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) I0927 22:56:07.913545 18464 raft_consensus.cc:2176] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 93ce3bfa2f2a4d00abf078a11b3b7fec for term 4 because replica is either leader or believes a valid leader to be alive. I0927 22:56:07.914165 18373 leader_election.cc:377] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 4 pre-election: Vote denied by peer 9fa6f72caa24460e896f673a01c0fcf2. Message: Invalid argument: T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 93ce3bfa2f2a4d00abf078a11b3b7fec for term 4 because replica is either leader or believes a valid leader to be alive. I0927 22:56:07.914247 18373 leader_election.cc:258] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. I0927 22:56:07.914448 19139 raft_consensus.cc:2700] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec: Snoozing failure detection for 0.155s (election complete) I0927 22:56:07.914527 19139 raft_consensus.cc:2455] T 00000000000000000000000000000000 P 93ce3bfa2f2a4d00abf078a11b3b7fec [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0927 22:56:07.928623 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.928098 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.928323 (+ 225us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.928402 (+ 79us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":87} W0927 22:56:07.928777 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 40 times. W0927 22:56:07.933622 17612 external_mini_cluster.cc:1058] Unable to check leaks on /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master pid 18362: Timed out: CheckLeaks RPC to 127.17.51.61:11081 timed out after 1.000s (SENT) I0927 22:56:07.933761 17612 external_mini_cluster.cc:1001] Killing /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master with pid 18362 I0927 22:56:07.949180 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.948656 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.948871 (+ 215us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.948952 (+ 81us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":83} W0927 22:56:07.949386 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 41 times. W0927 22:56:07.973700 18436 connection.cc:513] server connection from 127.0.0.1:42666 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:07.973780 18285 connection.cc:513] client connection to 127.17.51.61:11081 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) W0927 22:56:07.974093 18436 connection.cc:513] client connection to 127.17.51.61:11081 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) I0927 22:56:07.974164 17612 external_mini_cluster.cc:1042] Attempting to check leaks for /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master pid 18427 W0927 22:56:07.974187 18436 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 93ce3bfa2f2a4d00abf078a11b3b7fec (127.17.51.61:11081): Couldn't send request to peer 93ce3bfa2f2a4d00abf078a11b3b7fec for tablet 00000000000000000000000000000000. Status: Network error: failed to read from TLS socket: Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. W0927 22:56:07.974408 18436 connection.cc:186] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0927 22:56:07.976701 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.976191 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.976422 (+ 231us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.976518 (+ 96us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":88} W0927 22:56:07.976852 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 42 times. I0927 22:56:07.993228 19147 sys_catalog.cc:335] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9fa6f72caa24460e896f673a01c0fcf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133b07c7153a40f0b5e18b033e7982c0" member_type: VOTER last_known_addr { host: "127.17.51.62" port: 11080 } } peers { permanent_uuid: "93ce3bfa2f2a4d00abf078a11b3b7fec" member_type: VOTER last_known_addr { host: "127.17.51.61" port: 11081 } } peers { permanent_uuid: "9fa6f72caa24460e896f673a01c0fcf2" member_type: VOTER last_known_addr { host: "127.17.51.60" port: 11082 } } } I0927 22:56:07.993502 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.992955 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.61:11081 0927 22:56:07.993266 (+ 311us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.993321 (+ 55us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":166} W0927 22:56:07.993652 18436 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 93ce3bfa2f2a4d00abf078a11b3b7fec (127.17.51.61:11081): Couldn't send request to peer 93ce3bfa2f2a4d00abf078a11b3b7fec for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.61:11081: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I0927 22:56:07.993505 19147 sys_catalog.cc:338] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 [sys.catalog]: This master's current role is: LEADER I0927 22:56:07.995261 19161 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.974781 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.60:11082 0927 22:56:07.975314 (+ 533us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.975332 (+ 18us) client_negotiation.cc:160] Beginning negotiation 0927 22:56:07.975444 (+ 112us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0927 22:56:07.975936 (+ 492us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0927 22:56:07.975942 (+ 6us) client_negotiation.cc:348] Received NEGOTIATE response from server 0927 22:56:07.977034 (+ 1092us) client_negotiation.cc:175] Negotiated authn=SASL 0927 22:56:07.977140 (+ 106us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:07.977199 (+ 59us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:07.977207 (+ 8us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:07.977627 (+ 420us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:07.977630 (+ 3us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:07.978034 (+ 404us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0927 22:56:07.978043 (+ 9us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0927 22:56:07.978807 (+ 764us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0927 22:56:07.978811 (+ 4us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0927 22:56:07.978894 (+ 83us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:07.979032 (+ 138us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0927 22:56:07.979034 (+ 2us) client_negotiation.cc:577] Calling sasl_client_start() 0927 22:56:07.979983 (+ 949us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0927 22:56:07.988769 (+ 8786us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:07.988773 (+ 4us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:07.988775 (+ 2us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:07.989040 (+ 265us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:07.993289 (+ 4249us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0927 22:56:07.993294 (+ 5us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0927 22:56:07.993298 (+ 4us) client_negotiation.cc:687] Calling sasl_client_step() 0927 22:56:07.993363 (+ 65us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0927 22:56:07.994436 (+ 1073us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0927 22:56:07.994439 (+ 3us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0927 22:56:07.994549 (+ 110us) client_negotiation.cc:695] Sending connection context 0927 22:56:07.994704 (+ 155us) client_negotiation.cc:225] Negotiation successful 0927 22:56:07.994907 (+ 203us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":257,"threads_started":1} I0927 22:56:07.995724 19162 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.974862 (+ 0us) reactor.cc:577] Submitting negotiation task for server connection from 127.0.0.1:42756 0927 22:56:07.975523 (+ 661us) server_negotiation.cc:176] Beginning negotiation 0927 22:56:07.975529 (+ 6us) server_negotiation.cc:365] Waiting for connection header 0927 22:56:07.975551 (+ 22us) server_negotiation.cc:373] Connection header received 0927 22:56:07.975645 (+ 94us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0927 22:56:07.975651 (+ 6us) server_negotiation.cc:412] Received NEGOTIATE request from client 0927 22:56:07.975752 (+ 101us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0927 22:56:07.975861 (+ 109us) server_negotiation.cc:197] Negotiated authn=SASL 0927 22:56:07.977337 (+ 1476us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:07.977503 (+ 166us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:07.978187 (+ 684us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0927 22:56:07.978666 (+ 479us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0927 22:56:07.978737 (+ 71us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0927 22:56:07.980227 (+ 1490us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0927 22:56:07.980249 (+ 22us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0927 22:56:07.980252 (+ 3us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0927 22:56:07.980277 (+ 25us) server_negotiation.cc:797] Calling sasl_server_start() 0927 22:56:07.982502 (+ 2225us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:07.992729 (+ 10227us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:07.992733 (+ 4us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:07.992735 (+ 2us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:07.993018 (+ 283us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0927 22:56:07.993718 (+ 700us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0927 22:56:07.993723 (+ 5us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0927 22:56:07.993726 (+ 3us) server_negotiation.cc:841] Calling sasl_server_step() 0927 22:56:07.993966 (+ 240us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0927 22:56:07.994435 (+ 469us) server_negotiation.cc:901] Waiting for connection context 0927 22:56:07.994836 (+ 401us) server_negotiation.cc:264] Negotiation successful 0927 22:56:07.995246 (+ 410us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":418,"threads_started":1} I0927 22:56:07.996155 19141 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0927 22:56:07.995699 (+ 0us) reactor.cc:577] Submitting negotiation task for client connection to 127.17.51.62:11080 0927 22:56:07.995900 (+ 201us) negotiation.cc:98] Waiting for socket to connect 0927 22:56:07.995966 (+ 66us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":93} I0927 22:56:07.996258 18484 generic_service.cc:169] Checking for leaks (request via RPC) W0927 22:56:07.996585 18437 consensus_peers.cc:435] T 00000000000000000000000000000000 P 9fa6f72caa24460e896f673a01c0fcf2 -> Peer 133b07c7153a40f0b5e18b033e7982c0 (127.17.51.62:11080): Couldn't send request to peer 133b07c7153a40f0b5e18b033e7982c0 for tablet 00000000000000000000000000000000. Status: Network error: Client connection negotiation failed: client connection to 127.17.51.62:11080: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 43 times. I0927 22:56:08.311009 17612 external_mini_cluster.cc:1001] Killing /tmp/dist-test-taskFgBoie/build/asan/bin/kudu-master with pid 18427 W0927 22:56:08.350648 18285 connection.cc:513] client connection to 127.17.51.60:11082 recv error: Network error: failed to read from TLS socket: Connection reset by peer (error 104) [ OK ] SecurityComponentsFaultsITest.KdcRestartsInTheMiddle (126829 ms) [----------] 2 tests from SecurityComponentsFaultsITest (250679 ms total) [----------] Global test environment tear-down [==========] 2 tests from 1 test case ran. (250679 ms total) [ PASSED ] 2 tests.