[==========] Running 23 tests from 3 test cases. [----------] Global test environment set-up. [----------] 9 tests from SecurityITest [ RUN ] SecurityITest.SmokeTestAsAuthorizedUser Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:30:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): setting up network... Jun 27 19:30:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:30:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): skipping unrecognized local address family 17 Jun 27 19:30:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): skipping unrecognized local address family 17 Jun 27 19:30:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): set up 1 sockets Jun 27 19:30:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): commencing operation krb5kdc: starting... W0627 19:30:42.498586 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.215s user 0.004s sys 0.017s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:30:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:30:42.742206 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:52465 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:30:42.965103 12973 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:30:42.969661 12973 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:30:42.970113 12973 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:30:42.989068 12973 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:30:42.989328 12973 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:30:42.989539 12973 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:30:42.989728 12973 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:30:42.989939 12973 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:30:42.990169 12973 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:30:43.025944 12973 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:52465 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:30:43.051932 12973 master_main.cc:80] Initializing master server... I0627 19:30:43.054203 12973 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 190592us Jun 27 19:30:43 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153043, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:30:43.060070 12973 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:30:43.075924 12973 fs_manager.cc:261] Metadata directory not provided I0627 19:30:43.076169 12973 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:30:43.076568 12973 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:30:43.076750 12973 server_base.cc:433] Attempting to create new FS layout instead I0627 19:30:43.091609 12973 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "3a64b25e668849de97960134d51d64c5" format_stamp: "Formatted at 2018-06-28 02:30:43 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:43.092803 12973 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "3a64b25e668849de97960134d51d64c5" format_stamp: "Formatted at 2018-06-28 02:30:43 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:43.119678 12973 fs_manager.cc:501] Time spent creating directory manager: real 0.026s user 0.013s sys 0.001s I0627 19:30:43.121621 12973 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:30:43.122241 12973 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:30:43.149112 12973 fs_manager.cc:417] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s I0627 19:30:43.149299 12973 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/wal uuid: "3a64b25e668849de97960134d51d64c5" format_stamp: "Formatted at 2018-06-28 02:30:43 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:43.149664 12973 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:30:43.446848 12973 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:30:43.454087 12973 master_main.cc:83] Starting Master server... I0627 19:30:43.541921 12973 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:52465 I0627 19:30:43.542999 12973 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:30:43.543115 12973 webserver.cc:186] Document root disabled I0627 19:30:43.544735 12973 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:34670/ I0627 19:30:43.545498 12973 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:30:43.548415 13239 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:30:43.551879 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 12973 WARNING: no policy specified for kudu/127.12.131.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.1@KRBTEST.COM" created. I0627 19:30:43.574292 13239 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5: Bootstrap starting. I0627 19:30:43.578249 13239 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5: No blocks or log segments found. Creating new log. I0627 19:30:43.579923 13239 log.cc:525] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5: Log is configured to *not* fsync() on all Append() calls I0627 19:30:43.588155 13239 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5: No bootstrap required, opened a new log Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. I0627 19:30:43.614285 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.12.131.1:0 --local_ip_for_outbound_sockets=127.12.131.1 --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:52465 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:30:43.614751 13239 raft_consensus.cc:325] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a64b25e668849de97960134d51d64c5" member_type: VOTER } I0627 19:30:43.615293 13239 raft_consensus.cc:351] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:30:43.615571 13239 raft_consensus.cc:628] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a64b25e668849de97960134d51d64c5, State: Initialized, Role: FOLLOWER I0627 19:30:43.616612 13239 consensus_queue.cc:227] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a64b25e668849de97960134d51d64c5" member_type: VOTER } I0627 19:30:43.618786 13239 raft_consensus.cc:365] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:30:43.619071 13239 raft_consensus.cc:457] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:30:43.619354 13239 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5: Snoozing failure detection for 1.992s (starting election) I0627 19:30:43.619663 13239 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:30:43.626785 13239 raft_consensus.cc:479] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a64b25e668849de97960134d51d64c5" member_type: VOTER } I0627 19:30:43.627604 13239 leader_election.cc:270] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:30:43.636459 13263 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5: Snoozing failure detection for 2.160s (election complete) I0627 19:30:43.638741 13263 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:30:43.645556 13263 raft_consensus.cc:593] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [term 1 LEADER]: Becoming Leader. State: Replica: 3a64b25e668849de97960134d51d64c5, State: Running, Role: LEADER I0627 19:30:43.646339 13263 consensus_queue.cc:204] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a64b25e668849de97960134d51d64c5" member_type: VOTER } I0627 19:30:43.658677 13264 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a64b25e668849de97960134d51d64c5. Latest consensus state: current_term: 1 leader_uuid: "3a64b25e668849de97960134d51d64c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a64b25e668849de97960134d51d64c5" member_type: VOTER } } I0627 19:30:43.659200 13264 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [sys.catalog]: This master's current role is: LEADER I0627 19:30:43.658893 13265 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a64b25e668849de97960134d51d64c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a64b25e668849de97960134d51d64c5" member_type: VOTER } } I0627 19:30:43.754104 13265 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [sys.catalog]: This master's current role is: LEADER I0627 19:30:43.660048 13239 sys_catalog.cc:337] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "3a64b25e668849de97960134d51d64c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a64b25e668849de97960134d51d64c5" member_type: VOTER } } I0627 19:30:43.767043 13239 sys_catalog.cc:340] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [sys.catalog]: This master's current role is: LEADER I0627 19:30:43.768344 13239 sys_catalog.cc:424] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:30:43.768844 13267 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:30:43.779721 12973 master_main.cc:86] Master server successfully started. I0627 19:30:43.791369 13267 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:30:43.837733 13262 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:30:43.838238 13262 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.1 W0627 19:30:43.838676 13262 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:30:43.855587 13262 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:30:43.855839 13262 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:30:43.856111 13262 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.1 W0627 19:30:43.856659 13262 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.1 I0627 19:30:43.878768 13267 catalog_manager.cc:904] Generated new certificate authority record I0627 19:30:43.881086 13267 catalog_manager.cc:1026] Loading token signing keys... I0627 19:30:43.904775 13267 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5: Generated new TSK 0 I0627 19:30:43.908970 13262 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.1:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:52465 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:30:43.923892 13262 tablet_server_main.cc:85] Initializing tablet server... I0627 19:30:43.926743 13262 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 190592us Jun 27 19:30:43 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153043, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:30:43.930070 13262 init.cc:339] Logged in from keytab as kudu/127.12.131.1@KRBTEST.COM (short username kudu) I0627 19:30:43.933934 13262 fs_manager.cc:261] Metadata directory not provided I0627 19:30:43.934229 13262 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:30:43.934700 13262 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:30:43.934897 13262 server_base.cc:433] Attempting to create new FS layout instead I0627 19:30:43.950666 13262 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/data/instance: uuid: "120265d660e54bdcadcdcd29e6fe205a" format_stamp: "Formatted at 2018-06-28 02:30:43 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:43.951740 13262 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/wal/instance: uuid: "120265d660e54bdcadcdcd29e6fe205a" format_stamp: "Formatted at 2018-06-28 02:30:43 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:43.975267 13262 fs_manager.cc:501] Time spent creating directory manager: real 0.023s user 0.013s sys 0.005s I0627 19:30:43.977248 13262 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:30:43.977890 13262 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:30:44.003484 13262 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.001s sys 0.005s I0627 19:30:44.003732 13262 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/wal uuid: "120265d660e54bdcadcdcd29e6fe205a" format_stamp: "Formatted at 2018-06-28 02:30:43 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:44.004092 13262 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:30:44.286191 13262 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:30:44.291107 13262 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:30:44.292729 13262 tablet_server_main.cc:90] Starting tablet server... I0627 19:30:44.713281 13262 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.1:34670 I0627 19:30:44.715046 13262 webserver.cc:175] Starting webserver on 127.12.131.1:0 I0627 19:30:44.715270 13262 webserver.cc:186] Document root disabled I0627 19:30:44.735558 13262 webserver.cc:313] Webserver started. Bound to: http://127.12.131.1:48872/ I0627 19:30:44.736680 13262 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/data/info.pb I0627 19:30:44.737306 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 13262 I0627 19:30:44.760818 13262 tablet_server_main.cc:93] Tablet server successfully started. WARNING: no policy specified for kudu/127.12.131.2@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.2@KRBTEST.COM" created. Jun 27 19:30:44 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153043, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. W0627 19:30:44.928746 12812 mini_kdc.cc:260] Time spent creating service keytab for kudu/127.12.131.2: real 0.191s user 0.002s sys 0.007s I0627 19:30:44.930164 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.12.131.2:0 --local_ip_for_outbound_sockets=127.12.131.2 --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:52465 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:30:44.981031 13664 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:44.750636 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:52588 0627 19:30:44.762399 (+ 11763us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:44.762439 (+ 40us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:44.762544 (+ 105us) server_negotiation.cc:373] Connection header received 0627 19:30:44.856502 (+ 93958us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:44.856547 (+ 45us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:44.856806 (+ 259us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:44.857160 (+ 354us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:44.871893 (+ 14733us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:44.872795 (+ 902us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:44.881670 (+ 8875us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:44.883411 (+ 1741us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:44.883627 (+ 216us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:44.927491 (+ 43864us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:44.927598 (+ 107us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:44.927630 (+ 32us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:30:44.927721 (+ 91us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:44.934656 (+ 6935us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:44.961241 (+ 26585us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:44.961275 (+ 34us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:44.961297 (+ 22us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:44.962097 (+ 800us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:44.963661 (+ 1564us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:44.963687 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:44.963706 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:44.964311 (+ 605us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:44.964687 (+ 376us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:44.978664 (+ 13977us) server_negotiation.cc:264] Negotiation successful 0627 19:30:44.979116 (+ 452us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":11446,"thread_start_us":9275,"threads_started":1} I0627 19:30:44.996174 13662 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:44.748650 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:52465 0627 19:30:44.755518 (+ 6868us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:44.755604 (+ 86us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:44.830036 (+ 74432us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:44.868495 (+ 38459us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:44.868529 (+ 34us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:44.869454 (+ 925us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:30:44.869757 (+ 303us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:44.870178 (+ 421us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:44.870223 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:44.873175 (+ 2952us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:44.873202 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:44.874429 (+ 1227us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:44.874479 (+ 50us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:44.884336 (+ 9857us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:44.884365 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:44.894837 (+ 10472us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:44.895162 (+ 325us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:30:44.895181 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:30:44.898580 (+ 3399us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:30:44.935708 (+ 37128us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:30:44.935736 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:30:44.935758 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:30:44.939128 (+ 3370us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:30:44.962808 (+ 23680us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:30:44.962834 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:30:44.962854 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:30:44.963123 (+ 269us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:30:44.973077 (+ 9954us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:30:44.973110 (+ 33us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:30:44.974694 (+ 1584us) client_negotiation.cc:695] Sending connection context 0627 19:30:44.975207 (+ 513us) client_negotiation.cc:225] Negotiation successful 0627 19:30:44.975664 (+ 457us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":6327,"thread_start_us":8275,"threads_started":1} I0627 19:30:45.003582 13658 heartbeater.cc:346] Connected to a master server at 127.12.131.62:52465 I0627 19:30:45.003917 13658 heartbeater.cc:426] Registering TS with master... I0627 19:30:45.004588 13658 heartbeater.cc:475] Master 127.12.131.62:52465 requested a full tablet report, sending... I0627 19:30:45.007254 13186 ts_manager.cc:98] Registered new tserver with Master: 120265d660e54bdcadcdcd29e6fe205a (127.12.131.1:34670) I0627 19:30:45.010051 13186 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.1@KRBTEST.COM'} at 127.12.131.1:52588 WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:30:45.422739 13710 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:30:45.423286 13710 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.2 W0627 19:30:45.423867 13710 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:30:45.440552 13710 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:30:45.440780 13710 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:30:45.441058 13710 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.2 W0627 19:30:45.441607 13710 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.2 I0627 19:30:45.467854 13710 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.2:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:52465 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.2 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:30:45.484082 13710 tablet_server_main.cc:85] Initializing tablet server... I0627 19:30:45.486470 13710 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 191592us Jun 27 19:30:45 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153045, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:30:45.489698 13710 init.cc:339] Logged in from keytab as kudu/127.12.131.2@KRBTEST.COM (short username kudu) I0627 19:30:45.497308 13710 fs_manager.cc:261] Metadata directory not provided I0627 19:30:45.497548 13710 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:30:45.497928 13710 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:30:45.498124 13710 server_base.cc:433] Attempting to create new FS layout instead I0627 19:30:45.512949 13710 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/data/instance: uuid: "e75537a5558b4f3395cdd0928d3daa51" format_stamp: "Formatted at 2018-06-28 02:30:45 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:45.514077 13710 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/wal/instance: uuid: "e75537a5558b4f3395cdd0928d3daa51" format_stamp: "Formatted at 2018-06-28 02:30:45 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:45.533967 13710 fs_manager.cc:501] Time spent creating directory manager: real 0.019s user 0.012s sys 0.004s I0627 19:30:45.536062 13710 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:30:45.536731 13710 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:30:45.553540 13710 fs_manager.cc:417] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s I0627 19:30:45.553856 13710 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/wal uuid: "e75537a5558b4f3395cdd0928d3daa51" format_stamp: "Formatted at 2018-06-28 02:30:45 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:45.554200 13710 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:30:46.014083 13658 heartbeater.cc:467] Master 127.12.131.62:52465 was elected leader, sending a full tablet report... I0627 19:30:46.066351 13710 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:30:46.090752 13710 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:30:46.102444 13710 tablet_server_main.cc:90] Starting tablet server... I0627 19:30:46.779698 13710 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.2:34174 I0627 19:30:46.780928 13710 webserver.cc:175] Starting webserver on 127.12.131.2:0 I0627 19:30:46.781035 13710 webserver.cc:186] Document root disabled I0627 19:30:46.782681 13710 webserver.cc:313] Webserver started. Bound to: http://127.12.131.2:58138/ I0627 19:30:46.783406 13710 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/data/info.pb I0627 19:30:46.789324 13710 tablet_server_main.cc:93] Tablet server successfully started. I0627 19:30:46.791366 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 13710 WARNING: no policy specified for kudu/127.12.131.3@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.3@KRBTEST.COM" created. Jun 27 19:30:46 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153045, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:30:46.836230 14083 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:46.787353 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:41071 0627 19:30:46.801419 (+ 14066us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:46.801444 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:46.801486 (+ 42us) server_negotiation.cc:373] Connection header received 0627 19:30:46.810032 (+ 8546us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:46.810054 (+ 22us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:46.810149 (+ 95us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:46.810271 (+ 122us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:46.812453 (+ 2182us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:46.812925 (+ 472us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:46.814547 (+ 1622us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:46.815168 (+ 621us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:46.815326 (+ 158us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:46.819440 (+ 4114us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:46.819512 (+ 72us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:46.819535 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:30:46.819617 (+ 82us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:46.824751 (+ 5134us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:46.826917 (+ 2166us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:46.826935 (+ 18us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:46.826945 (+ 10us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:46.827321 (+ 376us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:46.828440 (+ 1119us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:46.828468 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:46.828487 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:46.828942 (+ 455us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:46.829269 (+ 327us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:46.834104 (+ 4835us) server_negotiation.cc:264] Negotiation successful 0627 19:30:46.834480 (+ 376us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":13839,"thread_start_us":12367,"threads_started":1} I0627 19:30:46.839753 14084 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:46.788176 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:52465 0627 19:30:46.798595 (+ 10419us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:46.798758 (+ 163us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:46.809688 (+ 10930us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:46.810592 (+ 904us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:46.810625 (+ 33us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:46.811509 (+ 884us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:30:46.811803 (+ 294us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:46.812166 (+ 363us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:46.812220 (+ 54us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:46.813199 (+ 979us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:46.813223 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:46.814304 (+ 1081us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:46.814345 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:46.815372 (+ 1027us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:46.815402 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:46.815713 (+ 311us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:46.815911 (+ 198us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:30:46.815925 (+ 14us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:30:46.818764 (+ 2839us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:30:46.825170 (+ 6406us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:30:46.825197 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:30:46.825220 (+ 23us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:30:46.826550 (+ 1330us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:30:46.827718 (+ 1168us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:30:46.827744 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:30:46.827764 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:30:46.828013 (+ 249us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:30:46.832559 (+ 4546us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:30:46.832591 (+ 32us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:30:46.832981 (+ 390us) client_negotiation.cc:695] Sending connection context 0627 19:30:46.836818 (+ 3837us) client_negotiation.cc:225] Negotiation successful 0627 19:30:46.837219 (+ 401us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":9650,"thread_start_us":8992,"threads_started":1} I0627 19:30:46.842645 14081 heartbeater.cc:346] Connected to a master server at 127.12.131.62:52465 I0627 19:30:46.842962 14081 heartbeater.cc:426] Registering TS with master... I0627 19:30:46.843621 14081 heartbeater.cc:475] Master 127.12.131.62:52465 requested a full tablet report, sending... I0627 19:30:46.845598 13186 ts_manager.cc:98] Registered new tserver with Master: e75537a5558b4f3395cdd0928d3daa51 (127.12.131.2:34174) I0627 19:30:46.847383 13186 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.2@KRBTEST.COM'} at 127.12.131.2:41071 Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. I0627 19:30:46.852752 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.12.131.3:0 --local_ip_for_outbound_sockets=127.12.131.3 --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:52465 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:30:47.090373 14093 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:30:47.090843 14093 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.3 W0627 19:30:47.091251 14093 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:30:47.105598 14093 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:30:47.105756 14093 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:30:47.105938 14093 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.3 W0627 19:30:47.106273 14093 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.3 I0627 19:30:47.152866 14093 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.3:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:52465 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.3 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:30:47.171267 14093 tablet_server_main.cc:85] Initializing tablet server... I0627 19:30:47.174048 14093 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 192592us Jun 27 19:30:47 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153047, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:30:47.177487 14093 init.cc:339] Logged in from keytab as kudu/127.12.131.3@KRBTEST.COM (short username kudu) I0627 19:30:47.182420 14093 fs_manager.cc:261] Metadata directory not provided I0627 19:30:47.182631 14093 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:30:47.183017 14093 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:30:47.183219 14093 server_base.cc:433] Attempting to create new FS layout instead I0627 19:30:47.200214 14093 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/data/instance: uuid: "21176505f7454e48a04bef7c77e66992" format_stamp: "Formatted at 2018-06-28 02:30:47 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:47.202144 14093 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/wal/instance: uuid: "21176505f7454e48a04bef7c77e66992" format_stamp: "Formatted at 2018-06-28 02:30:47 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:47.218785 14093 fs_manager.cc:501] Time spent creating directory manager: real 0.016s user 0.012s sys 0.003s I0627 19:30:47.220590 14093 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:30:47.221194 14093 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:30:47.237053 14093 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0627 19:30:47.237294 14093 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/wal uuid: "21176505f7454e48a04bef7c77e66992" format_stamp: "Formatted at 2018-06-28 02:30:47 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:47.237691 14093 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:30:47.649133 14093 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:30:47.657105 14093 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:30:47.663493 14093 tablet_server_main.cc:90] Starting tablet server... I0627 19:30:47.851248 14081 heartbeater.cc:467] Master 127.12.131.62:52465 was elected leader, sending a full tablet report... I0627 19:30:48.142465 14093 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.3:53767 I0627 19:30:48.143903 14093 webserver.cc:175] Starting webserver on 127.12.131.3:0 I0627 19:30:48.144024 14093 webserver.cc:186] Document root disabled I0627 19:30:48.151499 14093 webserver.cc:313] Webserver started. Bound to: http://127.12.131.3:41861/ I0627 19:30:48.152611 14093 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/data/info.pb I0627 19:30:48.162761 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 14093 I0627 19:30:48.168350 14093 tablet_server_main.cc:93] Tablet server successfully started. Jun 27 19:30:48 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153047, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM Jun 27 19:30:48 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:30:48.222163 14574 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.161012 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:52465 0627 19:30:48.162864 (+ 1852us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:48.162939 (+ 75us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:48.190124 (+ 27185us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:48.191231 (+ 1107us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:48.191263 (+ 32us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:48.192119 (+ 856us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:30:48.192447 (+ 328us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.193844 (+ 1397us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.193896 (+ 52us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.195038 (+ 1142us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.195064 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.196194 (+ 1130us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.196238 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.198130 (+ 1892us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.198158 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.198497 (+ 339us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.198767 (+ 270us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:30:48.198788 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:30:48.202080 (+ 3292us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:30:48.211486 (+ 9406us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:30:48.211519 (+ 33us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:30:48.211547 (+ 28us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:30:48.212914 (+ 1367us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:30:48.216388 (+ 3474us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:30:48.216414 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:30:48.216434 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:30:48.216714 (+ 280us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:30:48.218299 (+ 1585us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:30:48.218327 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:30:48.218785 (+ 458us) client_negotiation.cc:695] Sending connection context 0627 19:30:48.219276 (+ 491us) client_negotiation.cc:225] Negotiation successful 0627 19:30:48.219704 (+ 428us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1501,"thread_start_us":8587,"threads_started":1} I0627 19:30:48.225296 14573 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.160264 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:35051 0627 19:30:48.176541 (+ 16277us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:48.176566 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:48.176606 (+ 40us) server_negotiation.cc:373] Connection header received 0627 19:30:48.190475 (+ 13869us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:48.190508 (+ 33us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:48.190636 (+ 128us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:48.190774 (+ 138us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:48.194113 (+ 3339us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.194657 (+ 544us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.196461 (+ 1804us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.197525 (+ 1064us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.197685 (+ 160us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.203430 (+ 5745us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:48.203503 (+ 73us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:48.203529 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:30:48.203611 (+ 82us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:48.210821 (+ 7210us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:48.213570 (+ 2749us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:48.213597 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:48.213616 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:48.215930 (+ 2314us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:48.217127 (+ 1197us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:48.217153 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:48.217173 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:48.217667 (+ 494us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:48.217982 (+ 315us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:48.223137 (+ 5155us) server_negotiation.cc:264] Negotiation successful 0627 19:30:48.223513 (+ 376us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":1521,"server-negotiator.queue_time_us":15060,"thread_start_us":14844,"threads_started":1} I0627 19:30:48.227507 14578 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.170803 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:58705 0627 19:30:48.176219 (+ 5416us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:48.176243 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:48.183135 (+ 6892us) server_negotiation.cc:373] Connection header received 0627 19:30:48.192606 (+ 9471us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:48.192636 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:48.192759 (+ 123us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:48.192893 (+ 134us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:48.196535 (+ 3642us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.196942 (+ 407us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.198901 (+ 1959us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.199642 (+ 741us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.199807 (+ 165us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.205341 (+ 5534us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:48.205368 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:48.205394 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:30:48.205500 (+ 106us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:48.217184 (+ 11684us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:48.219395 (+ 2211us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:48.219423 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:48.219441 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:48.220087 (+ 646us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:48.220827 (+ 740us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:48.220858 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:48.220877 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:48.221342 (+ 465us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:48.224552 (+ 3210us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:48.224769 (+ 217us) server_negotiation.cc:264] Negotiation successful 0627 19:30:48.225306 (+ 537us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":3164,"server-negotiator.queue_time_us":5209,"thread_start_us":4637,"threads_started":1} I0627 19:30:48.227706 14569 heartbeater.cc:346] Connected to a master server at 127.12.131.62:52465 I0627 19:30:48.228058 14569 heartbeater.cc:426] Registering TS with master... I0627 19:30:48.228684 14569 heartbeater.cc:475] Master 127.12.131.62:52465 requested a full tablet report, sending... I0627 19:30:48.230743 13186 ts_manager.cc:98] Registered new tserver with Master: 21176505f7454e48a04bef7c77e66992 (127.12.131.3:53767) I0627 19:30:48.232213 13186 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.3@KRBTEST.COM'} at 127.12.131.3:35051 I0627 19:30:48.234652 12812 external_mini_cluster.cc:490] 3 TS(s) registered with all masters Jun 27 19:30:48 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-user@KRBTEST.COM: Jun 27 19:30:48 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:30:48.349589 14578 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.322943 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:58709 0627 19:30:48.323280 (+ 337us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:48.323305 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:48.326927 (+ 3622us) server_negotiation.cc:373] Connection header received 0627 19:30:48.327577 (+ 650us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:48.327605 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:48.327738 (+ 133us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:48.327870 (+ 132us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:48.330391 (+ 2521us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.330850 (+ 459us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.336090 (+ 5240us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.336834 (+ 744us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.337024 (+ 190us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.340143 (+ 3119us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:48.340173 (+ 30us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:48.340197 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:30:48.340275 (+ 78us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:48.344420 (+ 4145us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:48.345661 (+ 1241us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:48.345679 (+ 18us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:48.345690 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:48.346074 (+ 384us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:48.346556 (+ 482us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:48.346576 (+ 20us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:48.346587 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:48.346893 (+ 306us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:48.347177 (+ 284us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:48.347723 (+ 546us) server_negotiation.cc:264] Negotiation successful 0627 19:30:48.348156 (+ 433us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":132} I0627 19:30:48.355975 13186 catalog_manager.cc:1339] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.12.131.62:58709: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "val" type: INT32 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0627 19:30:48.370903 13186 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0627 19:30:48.427825 14659 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.403456 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:37911 0627 19:30:48.409392 (+ 5936us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:48.409425 (+ 33us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:48.412569 (+ 3144us) server_negotiation.cc:373] Connection header received 0627 19:30:48.416241 (+ 3672us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:48.416272 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:48.416587 (+ 315us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:48.416746 (+ 159us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:30:48.417980 (+ 1234us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.418534 (+ 554us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.421339 (+ 2805us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.423732 (+ 2393us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.423917 (+ 185us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.426211 (+ 2294us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:48.426347 (+ 136us) server_negotiation.cc:264] Negotiation successful 0627 19:30:48.426638 (+ 291us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5688,"thread_start_us":5382,"threads_started":1} I0627 19:30:48.429455 14657 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.401919 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.2:34174 0627 19:30:48.411977 (+ 10058us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:48.412103 (+ 126us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:48.412335 (+ 232us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:48.417043 (+ 4708us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:48.417063 (+ 20us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:48.417105 (+ 42us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:30:48.417474 (+ 369us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.417707 (+ 233us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.417749 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.418871 (+ 1122us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.418900 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.421036 (+ 2136us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.421081 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.424546 (+ 3465us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.424576 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.424893 (+ 317us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.425018 (+ 125us) client_negotiation.cc:695] Sending connection context 0627 19:30:48.425148 (+ 130us) client_negotiation.cc:225] Negotiation successful 0627 19:30:48.425451 (+ 303us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":9685,"thread_start_us":9839,"threads_started":1} I0627 19:30:48.434015 13955 tablet_service.cc:759] Processing CreateTablet for tablet 66a50e86b4dd4eeabb14833145c283f1 (table=test-table [id=7e71c095dcbe48d4a82397e05cf0dde6]), partition=RANGE (key) PARTITION UNBOUNDED I0627 19:30:48.435034 13955 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 66a50e86b4dd4eeabb14833145c283f1. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:30:48.445824 14662 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.412894 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:52133 0627 19:30:48.416738 (+ 3844us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:48.416768 (+ 30us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:48.423274 (+ 6506us) server_negotiation.cc:373] Connection header received 0627 19:30:48.429922 (+ 6648us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:48.429953 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:48.430167 (+ 214us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:48.430347 (+ 180us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:30:48.433170 (+ 2823us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.434045 (+ 875us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.438374 (+ 4329us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.440973 (+ 2599us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.441167 (+ 194us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.441532 (+ 365us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:48.444216 (+ 2684us) server_negotiation.cc:264] Negotiation successful 0627 19:30:48.444514 (+ 298us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":3614,"thread_start_us":2364,"threads_started":1} I0627 19:30:48.448034 14663 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.412778 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.3:53767 0627 19:30:48.423145 (+ 10367us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:48.423187 (+ 42us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:48.428963 (+ 5776us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:48.430784 (+ 1821us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:48.430813 (+ 29us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:48.430951 (+ 138us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:30:48.432592 (+ 1641us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.432921 (+ 329us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.432966 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.434321 (+ 1355us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.434348 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.438069 (+ 3721us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.438113 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.441768 (+ 3655us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.441798 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.442106 (+ 308us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.442218 (+ 112us) client_negotiation.cc:695] Sending connection context 0627 19:30:48.446235 (+ 4017us) client_negotiation.cc:225] Negotiation successful 0627 19:30:48.446541 (+ 306us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":10163,"mutex_wait_us":1273,"spinlock_wait_cycles":12544,"thread_start_us":10499,"threads_started":1} I0627 19:30:48.451138 14445 tablet_service.cc:759] Processing CreateTablet for tablet 66a50e86b4dd4eeabb14833145c283f1 (table=test-table [id=7e71c095dcbe48d4a82397e05cf0dde6]), partition=RANGE (key) PARTITION UNBOUNDED I0627 19:30:48.452164 14445 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 66a50e86b4dd4eeabb14833145c283f1. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:30:48.456068 14657 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.425842 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.1:34670 0627 19:30:48.429915 (+ 4073us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:48.429949 (+ 34us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:48.430081 (+ 132us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:48.430775 (+ 694us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:48.430801 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:48.430856 (+ 55us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:30:48.431112 (+ 256us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.431309 (+ 197us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.431353 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.442368 (+ 11015us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.442419 (+ 51us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.444068 (+ 1649us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.444112 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.453626 (+ 9514us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.453655 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.453965 (+ 310us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.454081 (+ 116us) client_negotiation.cc:695] Sending connection context 0627 19:30:48.454189 (+ 108us) client_negotiation.cc:225] Negotiation successful 0627 19:30:48.454655 (+ 466us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":3886,"mutex_wait_us":84,"spinlock_wait_cycles":271744,"thread_start_us":5139,"threads_started":1} I0627 19:30:48.458567 14665 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.424033 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:41461 0627 19:30:48.425820 (+ 1787us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:48.425851 (+ 31us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:48.430039 (+ 4188us) server_negotiation.cc:373] Connection header received 0627 19:30:48.430283 (+ 244us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:48.430312 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:48.430560 (+ 248us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:48.430723 (+ 163us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:30:48.431969 (+ 1246us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.432564 (+ 595us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.444598 (+ 12034us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.450307 (+ 5709us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.456552 (+ 6245us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.456910 (+ 358us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:48.457047 (+ 137us) server_negotiation.cc:264] Negotiation successful 0627 19:30:48.457337 (+ 290us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1557,"thread_start_us":1273,"threads_started":1} I0627 19:30:48.461483 13518 tablet_service.cc:759] Processing CreateTablet for tablet 66a50e86b4dd4eeabb14833145c283f1 (table=test-table [id=7e71c095dcbe48d4a82397e05cf0dde6]), partition=RANGE (key) PARTITION UNBOUNDED I0627 19:30:48.462558 13518 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 66a50e86b4dd4eeabb14833145c283f1. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:30:48.485455 13955 ts_tablet_manager.cc:1173] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Registered tablet (data state: TABLET_DATA_READY) I0627 19:30:48.492810 14445 ts_tablet_manager.cc:1173] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: Registered tablet (data state: TABLET_DATA_READY) I0627 19:30:48.499006 14674 ts_tablet_manager.cc:1028] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: Bootstrapping tablet I0627 19:30:48.501621 14674 tablet_bootstrap.cc:427] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: Bootstrap starting. I0627 19:30:48.505041 13518 ts_tablet_manager.cc:1173] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Registered tablet (data state: TABLET_DATA_READY) I0627 19:30:48.502763 14675 ts_tablet_manager.cc:1028] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Bootstrapping tablet I0627 19:30:48.507431 14674 tablet_bootstrap.cc:578] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: No blocks or log segments found. Creating new log. I0627 19:30:48.508199 14675 tablet_bootstrap.cc:427] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Bootstrap starting. I0627 19:30:48.511068 14674 log.cc:525] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: Log is configured to *not* fsync() on all Append() calls I0627 19:30:48.513900 14675 tablet_bootstrap.cc:578] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: No blocks or log segments found. Creating new log. I0627 19:30:48.515676 14675 log.cc:525] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Log is configured to *not* fsync() on all Append() calls I0627 19:30:48.518424 14676 ts_tablet_manager.cc:1028] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Bootstrapping tablet I0627 19:30:48.520071 14676 tablet_bootstrap.cc:427] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Bootstrap starting. I0627 19:30:48.524065 14676 tablet_bootstrap.cc:578] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: No blocks or log segments found. Creating new log. I0627 19:30:48.525871 14674 tablet_bootstrap.cc:427] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: No bootstrap required, opened a new log I0627 19:30:48.526309 14674 ts_tablet_manager.cc:1045] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.003s I0627 19:30:48.526268 14676 log.cc:525] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Log is configured to *not* fsync() on all Append() calls I0627 19:30:48.527936 14675 tablet_bootstrap.cc:427] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: No bootstrap required, opened a new log I0627 19:30:48.528424 14675 ts_tablet_manager.cc:1045] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.002s I0627 19:30:48.545310 14676 tablet_bootstrap.cc:427] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: No bootstrap required, opened a new log I0627 19:30:48.546174 14676 ts_tablet_manager.cc:1045] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.006s I0627 19:30:48.547878 14674 raft_consensus.cc:325] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75537a5558b4f3395cdd0928d3daa51" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 34174 } } peers { permanent_uuid: "21176505f7454e48a04bef7c77e66992" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 53767 } } peers { permanent_uuid: "120265d660e54bdcadcdcd29e6fe205a" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 34670 } } I0627 19:30:48.548981 14674 raft_consensus.cc:351] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:30:48.549232 14674 raft_consensus.cc:628] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21176505f7454e48a04bef7c77e66992, State: Initialized, Role: FOLLOWER I0627 19:30:48.550292 14674 consensus_queue.cc:227] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75537a5558b4f3395cdd0928d3daa51" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 34174 } } peers { permanent_uuid: "21176505f7454e48a04bef7c77e66992" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 53767 } } peers { permanent_uuid: "120265d660e54bdcadcdcd29e6fe205a" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 34670 } } I0627 19:30:48.559770 14569 heartbeater.cc:467] Master 127.12.131.62:52465 was elected leader, sending a full tablet report... I0627 19:30:48.562116 14674 ts_tablet_manager.cc:1073] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: Time spent starting tablet: real 0.036s user 0.019s sys 0.008s I0627 19:30:48.569419 14675 raft_consensus.cc:325] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75537a5558b4f3395cdd0928d3daa51" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 34174 } } peers { permanent_uuid: "21176505f7454e48a04bef7c77e66992" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 53767 } } peers { permanent_uuid: "120265d660e54bdcadcdcd29e6fe205a" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 34670 } } I0627 19:30:48.572254 14675 raft_consensus.cc:351] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:30:48.573312 14675 raft_consensus.cc:628] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75537a5558b4f3395cdd0928d3daa51, State: Initialized, Role: FOLLOWER I0627 19:30:48.578979 14676 raft_consensus.cc:325] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75537a5558b4f3395cdd0928d3daa51" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 34174 } } peers { permanent_uuid: "21176505f7454e48a04bef7c77e66992" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 53767 } } peers { permanent_uuid: "120265d660e54bdcadcdcd29e6fe205a" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 34670 } } I0627 19:30:48.574417 14675 consensus_queue.cc:227] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75537a5558b4f3395cdd0928d3daa51" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 34174 } } peers { permanent_uuid: "21176505f7454e48a04bef7c77e66992" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 53767 } } peers { permanent_uuid: "120265d660e54bdcadcdcd29e6fe205a" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 34670 } } I0627 19:30:48.582492 14676 raft_consensus.cc:351] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:30:48.583155 14676 raft_consensus.cc:628] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120265d660e54bdcadcdcd29e6fe205a, State: Initialized, Role: FOLLOWER I0627 19:30:48.583779 14676 consensus_queue.cc:227] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75537a5558b4f3395cdd0928d3daa51" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 34174 } } peers { permanent_uuid: "21176505f7454e48a04bef7c77e66992" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 53767 } } peers { permanent_uuid: "120265d660e54bdcadcdcd29e6fe205a" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 34670 } } I0627 19:30:48.598024 14675 ts_tablet_manager.cc:1073] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Time spent starting tablet: real 0.069s user 0.019s sys 0.009s I0627 19:30:48.602713 14676 ts_tablet_manager.cc:1073] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Time spent starting tablet: real 0.056s user 0.020s sys 0.004s I0627 19:30:48.630259 14691 raft_consensus.cc:457] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0627 19:30:48.811744 14691 raft_consensus.cc:2730] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Snoozing failure detection for 1.927s (starting election) I0627 19:30:48.812064 14691 raft_consensus.cc:479] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75537a5558b4f3395cdd0928d3daa51" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 34174 } } peers { permanent_uuid: "21176505f7454e48a04bef7c77e66992" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 53767 } } peers { permanent_uuid: "120265d660e54bdcadcdcd29e6fe205a" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 34670 } } I0627 19:30:48.813498 14691 leader_election.cc:243] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e75537a5558b4f3395cdd0928d3daa51 (127.12.131.2:34174) I0627 19:30:48.814319 14691 leader_election.cc:243] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 21176505f7454e48a04bef7c77e66992 (127.12.131.3:53767) I0627 19:30:48.838698 14690 raft_consensus.cc:457] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0627 19:30:48.839010 14690 raft_consensus.cc:2730] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Snoozing failure detection for 2.238s (starting election) I0627 19:30:48.839325 14690 raft_consensus.cc:479] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75537a5558b4f3395cdd0928d3daa51" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 34174 } } peers { permanent_uuid: "21176505f7454e48a04bef7c77e66992" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 53767 } } peers { permanent_uuid: "120265d660e54bdcadcdcd29e6fe205a" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 34670 } } I0627 19:30:48.840840 14690 leader_election.cc:243] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 21176505f7454e48a04bef7c77e66992 (127.12.131.3:53767) I0627 19:30:48.841480 14690 leader_election.cc:243] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 120265d660e54bdcadcdcd29e6fe205a (127.12.131.1:34670) W0627 19:30:48.762684 13665 tablet.cc:1678] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:30:48.877534 14659 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.816177 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:43596 0627 19:30:48.816509 (+ 332us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:48.816533 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:48.861427 (+ 44894us) server_negotiation.cc:373] Connection header received 0627 19:30:48.861680 (+ 253us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:48.861708 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:48.861833 (+ 125us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:48.861945 (+ 112us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:30:48.868038 (+ 6093us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.868495 (+ 457us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.872520 (+ 4025us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.874181 (+ 1661us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.874351 (+ 170us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.874644 (+ 293us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:48.876389 (+ 1745us) server_negotiation.cc:264] Negotiation successful 0627 19:30:48.876497 (+ 108us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":89} W0627 19:30:48.667696 14572 tablet.cc:1678] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:30:48.880262 14693 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.815412 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.2:34174 0627 19:30:48.861241 (+ 45829us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:48.861293 (+ 52us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:48.861481 (+ 188us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:48.862978 (+ 1497us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:48.863002 (+ 24us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:48.863052 (+ 50us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:30:48.865713 (+ 2661us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.865954 (+ 241us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.865994 (+ 40us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.868758 (+ 2764us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.868784 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.872183 (+ 3399us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.872228 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.874864 (+ 2636us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.874892 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.875647 (+ 755us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.875802 (+ 155us) client_negotiation.cc:695] Sending connection context 0627 19:30:48.878459 (+ 2657us) client_negotiation.cc:225] Negotiation successful 0627 19:30:48.878520 (+ 61us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":45554,"thread_start_us":45246,"threads_started":1} I0627 19:30:48.884168 14012 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "66a50e86b4dd4eeabb14833145c283f1" candidate_uuid: "120265d660e54bdcadcdcd29e6fe205a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75537a5558b4f3395cdd0928d3daa51" is_pre_election: true I0627 19:30:48.884812 14012 raft_consensus.cc:2730] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Snoozing failure detection for 1.959s (vote granted) I0627 19:30:48.885071 14012 raft_consensus.cc:2243] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 120265d660e54bdcadcdcd29e6fe205a in term 0. I0627 19:30:48.887264 13343 leader_election.cc:379] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [CANDIDATE]: Term 1 pre-election: Vote granted by peer e75537a5558b4f3395cdd0928d3daa51 (127.12.131.2:34174) I0627 19:30:48.887620 13343 leader_election.cc:270] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0627 19:30:48.888536 14691 raft_consensus.cc:2730] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Snoozing failure detection for 1.509s (election complete) I0627 19:30:48.888826 14691 raft_consensus.cc:2537] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 0 FOLLOWER]: Leader pre-election won for term 1 I0627 19:30:48.889040 14691 raft_consensus.cc:457] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0627 19:30:48.889225 14691 raft_consensus.cc:2730] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Snoozing failure detection for 2.162s (starting election) I0627 19:30:48.889422 14691 raft_consensus.cc:2782] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 0 FOLLOWER]: Advancing to term 1 I0627 19:30:48.897220 14691 raft_consensus.cc:479] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75537a5558b4f3395cdd0928d3daa51" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 34174 } } peers { permanent_uuid: "21176505f7454e48a04bef7c77e66992" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 53767 } } peers { permanent_uuid: "120265d660e54bdcadcdcd29e6fe205a" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 34670 } } I0627 19:30:48.898805 14691 leader_election.cc:243] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [CANDIDATE]: Term 1 election: Requesting vote from peer e75537a5558b4f3395cdd0928d3daa51 (127.12.131.2:34174) I0627 19:30:48.904395 14691 leader_election.cc:243] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [CANDIDATE]: Term 1 election: Requesting vote from peer 21176505f7454e48a04bef7c77e66992 (127.12.131.3:53767) I0627 19:30:48.905915 14012 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "66a50e86b4dd4eeabb14833145c283f1" candidate_uuid: "120265d660e54bdcadcdcd29e6fe205a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75537a5558b4f3395cdd0928d3daa51" I0627 19:30:48.906388 14012 raft_consensus.cc:2782] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:30:48.906702 14012 raft_consensus.cc:2730] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Snoozing failure detection for 2.206s (vote granted) I0627 19:30:48.912595 14012 raft_consensus.cc:2243] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 120265d660e54bdcadcdcd29e6fe205a in term 1. I0627 19:30:48.903993 14696 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.862524 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.3:53767 0627 19:30:48.873073 (+ 10549us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:48.873117 (+ 44us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:48.873246 (+ 129us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:48.882653 (+ 9407us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:48.882679 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:48.882739 (+ 60us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:30:48.883105 (+ 366us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.883484 (+ 379us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.883525 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.889524 (+ 5999us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.889542 (+ 18us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.898073 (+ 8531us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.898127 (+ 54us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.901741 (+ 3614us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.901767 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.902074 (+ 307us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.902230 (+ 156us) client_negotiation.cc:695] Sending connection context 0627 19:30:48.902505 (+ 275us) client_negotiation.cc:225] Negotiation successful 0627 19:30:48.902569 (+ 64us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":10340,"mutex_wait_us":56,"thread_start_us":10804,"threads_started":1} I0627 19:30:48.913714 13343 leader_election.cc:379] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [CANDIDATE]: Term 1 election: Vote granted by peer e75537a5558b4f3395cdd0928d3daa51 (127.12.131.2:34174) I0627 19:30:48.913971 13343 leader_election.cc:270] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:30:48.914983 14695 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.861543 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:57592 0627 19:30:48.881009 (+ 19466us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:48.881034 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:48.881073 (+ 39us) server_negotiation.cc:373] Connection header received 0627 19:30:48.881182 (+ 109us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:48.881202 (+ 20us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:48.881314 (+ 112us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:48.881451 (+ 137us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:30:48.887763 (+ 6312us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.888258 (+ 495us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.899136 (+ 10878us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.900982 (+ 1846us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.901224 (+ 242us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.901537 (+ 313us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:48.912747 (+ 11210us) server_negotiation.cc:264] Negotiation successful 0627 19:30:48.912834 (+ 87us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":19264,"thread_start_us":18146,"threads_started":1} I0627 19:30:48.917172 14490 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "66a50e86b4dd4eeabb14833145c283f1" candidate_uuid: "120265d660e54bdcadcdcd29e6fe205a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21176505f7454e48a04bef7c77e66992" is_pre_election: true I0627 19:30:48.917551 14489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "66a50e86b4dd4eeabb14833145c283f1" candidate_uuid: "120265d660e54bdcadcdcd29e6fe205a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21176505f7454e48a04bef7c77e66992" I0627 19:30:48.918169 14490 raft_consensus.cc:2730] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: Snoozing failure detection for 2.175s (vote granted) I0627 19:30:48.918473 14691 raft_consensus.cc:2730] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Snoozing failure detection for 1.682s (election complete) I0627 19:30:48.918458 14490 raft_consensus.cc:2243] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 120265d660e54bdcadcdcd29e6fe205a in term 0. I0627 19:30:48.918769 14691 raft_consensus.cc:2537] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:30:48.919416 14489 raft_consensus.cc:2218] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992 [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate 120265d660e54bdcadcdcd29e6fe205a for term 1 because replica is already servicing an update from a current leader or another vote. I0627 19:30:48.919983 13343 leader_election.cc:379] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [CANDIDATE]: Term 1 pre-election: Vote granted by peer 21176505f7454e48a04bef7c77e66992 (127.12.131.3:53767) I0627 19:30:48.921337 13343 leader_election.cc:393] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [CANDIDATE]: Term 1 election: Vote denied by peer 21176505f7454e48a04bef7c77e66992 (127.12.131.3:53767). Message: Service unavailable: T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992 [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate 120265d660e54bdcadcdcd29e6fe205a for term 1 because replica is already servicing an update from a current leader or another vote. I0627 19:30:48.927583 14691 raft_consensus.cc:593] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 1 LEADER]: Becoming Leader. State: Replica: 120265d660e54bdcadcdcd29e6fe205a, State: Running, Role: LEADER I0627 19:30:48.928153 14691 consensus_queue.cc:204] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75537a5558b4f3395cdd0928d3daa51" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 34174 } } peers { permanent_uuid: "21176505f7454e48a04bef7c77e66992" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 53767 } } peers { permanent_uuid: "120265d660e54bdcadcdcd29e6fe205a" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 34670 } } I0627 19:30:48.945801 13185 catalog_manager.cc:3798] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a reported cstate change: term changed from 0 to 1, leader changed from to 120265d660e54bdcadcdcd29e6fe205a (127.12.131.1). New cstate: current_term: 1 leader_uuid: "120265d660e54bdcadcdcd29e6fe205a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e75537a5558b4f3395cdd0928d3daa51" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 34174 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21176505f7454e48a04bef7c77e66992" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 53767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "120265d660e54bdcadcdcd29e6fe205a" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 34670 } health_report { overall_health: HEALTHY } } } W0627 19:30:48.794390 14082 tablet.cc:1678] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:30:49.000197 14662 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.843641 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:38606 0627 19:30:48.843971 (+ 330us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:48.843993 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:48.991213 (+147220us) server_negotiation.cc:373] Connection header received 0627 19:30:48.991456 (+ 243us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:48.991482 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:48.991604 (+ 122us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:48.991723 (+ 119us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:30:48.992854 (+ 1131us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.993413 (+ 559us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.996142 (+ 2729us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.997788 (+ 1646us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.997973 (+ 185us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.998219 (+ 246us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:48.998985 (+ 766us) server_negotiation.cc:264] Negotiation successful 0627 19:30:48.999053 (+ 68us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":127} I0627 19:30:49.000365 14694 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.842390 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.3:53767 0627 19:30:48.991076 (+148686us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:48.991121 (+ 45us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:48.991249 (+ 128us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:48.991773 (+ 524us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:48.991800 (+ 27us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:48.991856 (+ 56us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:30:48.992178 (+ 322us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.992653 (+ 475us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.992697 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.993609 (+ 912us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.993635 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.995895 (+ 2260us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.995944 (+ 49us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.998025 (+ 2081us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.998053 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.998354 (+ 301us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.998538 (+ 184us) client_negotiation.cc:695] Sending connection context 0627 19:30:48.998784 (+ 246us) client_negotiation.cc:225] Negotiation successful 0627 19:30:48.998843 (+ 59us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":146477,"mutex_wait_us":121,"spinlock_wait_cycles":16640,"thread_start_us":146162,"threads_started":1} I0627 19:30:49.000674 14704 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.989307 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:50573 0627 19:30:48.991262 (+ 1955us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:48.991285 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:48.991310 (+ 25us) server_negotiation.cc:373] Connection header received 0627 19:30:48.991438 (+ 128us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:48.991464 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:48.991587 (+ 123us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:48.991699 (+ 112us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:30:48.992778 (+ 1079us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.993398 (+ 620us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.995294 (+ 1896us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.996513 (+ 1219us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.996692 (+ 179us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.996941 (+ 249us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:48.999482 (+ 2541us) server_negotiation.cc:264] Negotiation successful 0627 19:30:48.999557 (+ 75us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1707,"thread_start_us":1482,"threads_started":1} I0627 19:30:49.001217 14703 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:48.989160 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.1:34670 0627 19:30:48.990763 (+ 1603us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:48.990804 (+ 41us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:48.990939 (+ 135us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:48.991989 (+ 1050us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:48.992008 (+ 19us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:48.992047 (+ 39us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:30:48.992297 (+ 250us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.992609 (+ 312us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.992637 (+ 28us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.993634 (+ 997us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.993651 (+ 17us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.995097 (+ 1446us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:48.995129 (+ 32us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:48.997132 (+ 2003us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:48.997158 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:48.997498 (+ 340us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:48.997656 (+ 158us) client_negotiation.cc:695] Sending connection context 0627 19:30:48.997907 (+ 251us) client_negotiation.cc:225] Negotiation successful 0627 19:30:48.997971 (+ 64us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1382,"mutex_wait_us":72,"thread_start_us":1078,"threads_started":1} I0627 19:30:49.001924 14489 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "66a50e86b4dd4eeabb14833145c283f1" candidate_uuid: "e75537a5558b4f3395cdd0928d3daa51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21176505f7454e48a04bef7c77e66992" is_pre_election: true I0627 19:30:49.002410 14489 raft_consensus.cc:2730] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: Snoozing failure detection for 1.772s (vote granted) I0627 19:30:49.002640 14489 raft_consensus.cc:2243] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e75537a5558b4f3395cdd0928d3daa51 in term 0. I0627 19:30:49.002730 13567 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "66a50e86b4dd4eeabb14833145c283f1" candidate_uuid: "e75537a5558b4f3395cdd0928d3daa51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "120265d660e54bdcadcdcd29e6fe205a" is_pre_election: true I0627 19:30:49.003345 13567 raft_consensus.cc:2204] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate e75537a5558b4f3395cdd0928d3daa51 for term 1 because replica is either leader or believes a valid leader to be alive. I0627 19:30:49.004318 13770 leader_election.cc:379] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 21176505f7454e48a04bef7c77e66992 (127.12.131.3:53767) I0627 19:30:49.004534 13770 leader_election.cc:270] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0627 19:30:49.004981 14690 raft_consensus.cc:2730] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Snoozing failure detection for 1.697s (election complete) I0627 19:30:49.005211 14690 raft_consensus.cc:2500] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won I0627 19:30:49.005472 13770 leader_election.cc:393] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 120265d660e54bdcadcdcd29e6fe205a (127.12.131.1:34670). Message: Invalid argument: T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate e75537a5558b4f3395cdd0928d3daa51 for term 1 because replica is either leader or believes a valid leader to be alive. I0627 19:30:49.053738 14704 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:49.045525 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:41468 0627 19:30:49.045796 (+ 271us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:49.045817 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:49.047784 (+ 1967us) server_negotiation.cc:373] Connection header received 0627 19:30:49.048072 (+ 288us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:49.048088 (+ 16us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:49.048199 (+ 111us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:49.048309 (+ 110us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:30:49.048966 (+ 657us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:49.049342 (+ 376us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:49.050585 (+ 1243us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:49.051209 (+ 624us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:49.051373 (+ 164us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:49.052054 (+ 681us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:30:49.052313 (+ 259us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:30:49.052435 (+ 122us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:49.052653 (+ 218us) server_negotiation.cc:264] Negotiation successful 0627 19:30:49.052816 (+ 163us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":70} I0627 19:30:49.062196 14489 raft_consensus.cc:2782] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:30:49.062744 14012 raft_consensus.cc:1081] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [term 1 FOLLOWER]: Refusing update from remote peer 120265d660e54bdcadcdcd29e6fe205a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0627 19:30:49.064576 14691 consensus_queue.cc:962] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e75537a5558b4f3395cdd0928d3daa51" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 34174 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0627 19:30:49.067730 14489 raft_consensus.cc:1081] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992 [term 1 FOLLOWER]: Refusing update from remote peer 120265d660e54bdcadcdcd29e6fe205a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0627 19:30:49.069315 14691 consensus_queue.cc:962] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21176505f7454e48a04bef7c77e66992" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 53767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0627 19:30:49.122007 13185 catalog_manager.cc:1731] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.12.131.62:58709: table { table_name: "test-table" } I0627 19:30:49.130182 13185 catalog_manager.cc:4037] T 00000000000000000000000000000000 P 3a64b25e668849de97960134d51d64c5: Sending DeleteTablet for 3 replicas of tablet 66a50e86b4dd4eeabb14833145c283f1 I0627 19:30:49.130586 13185 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 66a50e86b4dd4eeabb14833145c283f1 on e75537a5558b4f3395cdd0928d3daa51 (127.12.131.2:34174) (Table deleted at 2018-06-27 19:30:49 PDT) I0627 19:30:49.131083 13185 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 66a50e86b4dd4eeabb14833145c283f1 on 21176505f7454e48a04bef7c77e66992 (127.12.131.3:53767) (Table deleted at 2018-06-27 19:30:49 PDT) I0627 19:30:49.132196 14445 tablet_service.cc:800] Processing DeleteTablet for tablet 66a50e86b4dd4eeabb14833145c283f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-27 19:30:49 PDT) from {username='kudu', principal='kudu/127.12.131.62@KRBTEST.COM'} at 127.12.131.3:52133 I0627 19:30:49.132249 13955 tablet_service.cc:800] Processing DeleteTablet for tablet 66a50e86b4dd4eeabb14833145c283f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-27 19:30:49 PDT) from {username='kudu', principal='kudu/127.12.131.62@KRBTEST.COM'} at 127.12.131.2:37911 I0627 19:30:49.133990 14724 tablet_replica.cc:261] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: stopping tablet replica I0627 19:30:49.134296 14724 maintenance_manager.cc:237] P e75537a5558b4f3395cdd0928d3daa51: Unregistered op CompactRowSetsOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.134575 14724 maintenance_manager.cc:237] P e75537a5558b4f3395cdd0928d3daa51: Unregistered op MinorDeltaCompactionOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.134810 14724 maintenance_manager.cc:237] P e75537a5558b4f3395cdd0928d3daa51: Unregistered op MajorDeltaCompactionOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.135043 14724 maintenance_manager.cc:237] P e75537a5558b4f3395cdd0928d3daa51: Unregistered op UndoDeltaBlockGCOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.135304 14724 maintenance_manager.cc:237] P e75537a5558b4f3395cdd0928d3daa51: Unregistered op FlushMRSOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.135543 14724 maintenance_manager.cc:237] P e75537a5558b4f3395cdd0928d3daa51: Unregistered op FlushDeltaMemStoresOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.135756 14724 maintenance_manager.cc:237] P e75537a5558b4f3395cdd0928d3daa51: Unregistered op LogGCOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.135996 14724 raft_consensus.cc:2040] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [term 1 FOLLOWER]: Raft consensus shutting down. I0627 19:30:49.136360 14724 raft_consensus.cc:2067] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51 [term 1 FOLLOWER]: Raft consensus is shut down! I0627 19:30:49.131837 13185 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 66a50e86b4dd4eeabb14833145c283f1 on 120265d660e54bdcadcdcd29e6fe205a (127.12.131.1:34670) (Table deleted at 2018-06-27 19:30:49 PDT) I0627 19:30:49.137495 13518 tablet_service.cc:800] Processing DeleteTablet for tablet 66a50e86b4dd4eeabb14833145c283f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-27 19:30:49 PDT) from {username='kudu', principal='kudu/127.12.131.62@KRBTEST.COM'} at 127.12.131.1:41461 I0627 19:30:49.142376 14724 ts_tablet_manager.cc:1369] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Deleting tablet data with delete state TABLET_DATA_DELETED I0627 19:30:49.142308 14725 tablet_replica.cc:261] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: stopping tablet replica I0627 19:30:49.143682 14725 maintenance_manager.cc:237] P 21176505f7454e48a04bef7c77e66992: Unregistered op CompactRowSetsOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.143901 14725 maintenance_manager.cc:237] P 21176505f7454e48a04bef7c77e66992: Unregistered op MinorDeltaCompactionOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.144085 14725 maintenance_manager.cc:237] P 21176505f7454e48a04bef7c77e66992: Unregistered op MajorDeltaCompactionOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.144268 14725 maintenance_manager.cc:237] P 21176505f7454e48a04bef7c77e66992: Unregistered op UndoDeltaBlockGCOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.144520 14725 maintenance_manager.cc:237] P 21176505f7454e48a04bef7c77e66992: Unregistered op FlushMRSOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.142184 14726 tablet_replica.cc:261] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: stopping tablet replica I0627 19:30:49.144697 14725 maintenance_manager.cc:237] P 21176505f7454e48a04bef7c77e66992: Unregistered op FlushDeltaMemStoresOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.144875 14725 maintenance_manager.cc:237] P 21176505f7454e48a04bef7c77e66992: Unregistered op LogGCOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.144910 14726 maintenance_manager.cc:237] P 120265d660e54bdcadcdcd29e6fe205a: Unregistered op CompactRowSetsOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.145078 14725 raft_consensus.cc:2040] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992 [term 1 FOLLOWER]: Raft consensus shutting down. I0627 19:30:49.145186 14726 maintenance_manager.cc:237] P 120265d660e54bdcadcdcd29e6fe205a: Unregistered op MinorDeltaCompactionOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.145467 14726 maintenance_manager.cc:237] P 120265d660e54bdcadcdcd29e6fe205a: Unregistered op MajorDeltaCompactionOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.145434 14725 raft_consensus.cc:2067] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992 [term 1 FOLLOWER]: Raft consensus is shut down! I0627 19:30:49.145690 14726 maintenance_manager.cc:237] P 120265d660e54bdcadcdcd29e6fe205a: Unregistered op UndoDeltaBlockGCOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.146824 14726 maintenance_manager.cc:237] P 120265d660e54bdcadcdcd29e6fe205a: Unregistered op FlushMRSOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.147048 14726 maintenance_manager.cc:237] P 120265d660e54bdcadcdcd29e6fe205a: Unregistered op FlushDeltaMemStoresOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.147258 14726 maintenance_manager.cc:237] P 120265d660e54bdcadcdcd29e6fe205a: Unregistered op LogGCOp(66a50e86b4dd4eeabb14833145c283f1) I0627 19:30:49.147523 14726 raft_consensus.cc:2040] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 1 LEADER]: Raft consensus shutting down. I0627 19:30:49.147768 14726 consensus_peers.cc:455] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a -> Peer 21176505f7454e48a04bef7c77e66992 (127.12.131.3:53767): Closing peer: 21176505f7454e48a04bef7c77e66992 I0627 19:30:49.148032 14726 consensus_peers.cc:455] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a -> Peer e75537a5558b4f3395cdd0928d3daa51 (127.12.131.2:34174): Closing peer: e75537a5558b4f3395cdd0928d3daa51 I0627 19:30:49.148660 14726 raft_consensus.cc:2067] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a [term 1 FOLLOWER]: Raft consensus is shut down! I0627 19:30:49.152519 14726 ts_tablet_manager.cc:1369] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Deleting tablet data with delete state TABLET_DATA_DELETED I0627 19:30:49.152582 14724 ts_tablet_manager.cc:1382] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0627 19:30:49.152684 14725 ts_tablet_manager.cc:1369] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: Deleting tablet data with delete state TABLET_DATA_DELETED I0627 19:30:49.153617 14724 log.cc:986] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-1/wal/wals/66a50e86b4dd4eeabb14833145c283f1 I0627 19:30:49.157121 14724 ts_tablet_manager.cc:1403] T 66a50e86b4dd4eeabb14833145c283f1 P e75537a5558b4f3395cdd0928d3daa51: Deleting consensus metadata I0627 19:30:49.160290 13113 catalog_manager.cc:3103] TS e75537a5558b4f3395cdd0928d3daa51 (127.12.131.2:34174): tablet 66a50e86b4dd4eeabb14833145c283f1 (table test-table [id=7e71c095dcbe48d4a82397e05cf0dde6]) successfully deleted I0627 19:30:49.165516 14725 ts_tablet_manager.cc:1382] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0627 19:30:49.165870 14725 log.cc:986] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-2/wal/wals/66a50e86b4dd4eeabb14833145c283f1 I0627 19:30:49.166616 14725 ts_tablet_manager.cc:1403] T 66a50e86b4dd4eeabb14833145c283f1 P 21176505f7454e48a04bef7c77e66992: Deleting consensus metadata I0627 19:30:49.169268 13113 catalog_manager.cc:3103] TS 21176505f7454e48a04bef7c77e66992 (127.12.131.3:53767): tablet 66a50e86b4dd4eeabb14833145c283f1 (table test-table [id=7e71c095dcbe48d4a82397e05cf0dde6]) successfully deleted Jun 27 19:30:49 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[12903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.131.1@KRBTEST.COM I0627 19:30:49.178303 14726 ts_tablet_manager.cc:1382] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0627 19:30:49.179466 14726 log.cc:986] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1530153042096250-12812/minicluster-data/ts-0/wal/wals/66a50e86b4dd4eeabb14833145c283f1 I0627 19:30:49.182446 14726 ts_tablet_manager.cc:1403] T 66a50e86b4dd4eeabb14833145c283f1 P 120265d660e54bdcadcdcd29e6fe205a: Deleting consensus metadata I0627 19:30:49.184916 13113 catalog_manager.cc:3103] TS 120265d660e54bdcadcdcd29e6fe205a (127.12.131.1:34670): tablet 66a50e86b4dd4eeabb14833145c283f1 (table test-table [id=7e71c095dcbe48d4a82397e05cf0dde6]) successfully deleted I0627 19:30:49.193738 14704 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:49.155139 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:41470 0627 19:30:49.155975 (+ 836us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:49.156000 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:49.160802 (+ 4802us) server_negotiation.cc:373] Connection header received 0627 19:30:49.161413 (+ 611us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:49.161442 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:49.161575 (+ 133us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:49.161710 (+ 135us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:49.163936 (+ 2226us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:49.164395 (+ 459us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:49.165932 (+ 1537us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:49.166981 (+ 1049us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:49.167167 (+ 186us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:49.171487 (+ 4320us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:49.171522 (+ 35us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:49.171549 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:30:49.171635 (+ 86us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:49.186222 (+ 14587us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:49.188475 (+ 2253us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:49.188509 (+ 34us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:49.188528 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:49.189243 (+ 715us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:49.189943 (+ 700us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:49.189971 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:49.189991 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:49.190593 (+ 602us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:49.191012 (+ 419us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:49.191540 (+ 528us) server_negotiation.cc:264] Negotiation successful 0627 19:30:49.191977 (+ 437us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":629} W0627 19:30:49.194923 13647 server_base.cc:564] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.12.131.1:41470 I0627 19:30:49.227391 14740 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:49.199872 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:58727 0627 19:30:49.200800 (+ 928us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:49.200826 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:49.202493 (+ 1667us) server_negotiation.cc:373] Connection header received 0627 19:30:49.202884 (+ 391us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:49.202910 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:49.203044 (+ 134us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:49.203199 (+ 155us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:49.205561 (+ 2362us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:49.206002 (+ 441us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:49.207537 (+ 1535us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:49.208644 (+ 1107us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:49.208828 (+ 184us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:49.211988 (+ 3160us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:49.212018 (+ 30us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:49.212043 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:30:49.212128 (+ 85us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:49.218896 (+ 6768us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:49.220974 (+ 2078us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:49.221005 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:49.221025 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:49.221726 (+ 701us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:49.222327 (+ 601us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:49.222354 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:49.222374 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:49.222888 (+ 514us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:49.223213 (+ 325us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:49.225031 (+ 1818us) server_negotiation.cc:264] Negotiation successful 0627 19:30:49.225514 (+ 483us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":583,"threads_started":1} W0627 19:30:49.228781 13185 server_base.cc:564] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.12.131.62:58727 I0627 19:30:49.230280 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 13262 W0627 19:30:49.260588 13118 connection.cc:512] server connection from 127.12.131.1:52588 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:52588): Connection reset by peer (error 104) W0627 19:30:49.261520 13770 connection.cc:512] client connection to 127.12.131.1:34670 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:34670): Connection reset by peer (error 104) W0627 19:30:49.262655 13780 connection.cc:512] server connection from 127.12.131.1:43596 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:43596): Connection reset by peer (error 104) I0627 19:30:49.263547 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 13710 W0627 19:30:49.264482 14273 connection.cc:512] server connection from 127.12.131.1:57592 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:57592): Connection reset by peer (error 104) W0627 19:30:49.300745 14320 connection.cc:512] server connection from 127.12.131.2:38606 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:38606): Connection reset by peer (error 104) W0627 19:30:49.300909 13113 connection.cc:512] server connection from 127.12.131.2:41071 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:41071): Connection reset by peer (error 104) I0627 19:30:49.301487 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 14093 W0627 19:30:49.326668 13118 connection.cc:512] server connection from 127.12.131.3:35051 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.3:35051): Connection reset by peer (error 104) I0627 19:30:49.329195 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 12973 [ OK ] SecurityITest.SmokeTestAsAuthorizedUser (7136 ms) [ RUN ] SecurityITest.TestNoKerberosCredentials Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:30:49 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): setting up network... Jun 27 19:30:49 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:30:49 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): skipping unrecognized local address family 17 Jun 27 19:30:49 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): skipping unrecognized local address family 17 Jun 27 19:30:49 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): set up 1 sockets Jun 27 19:30:49 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): commencing operation krb5kdc: starting... W0627 19:30:49.481030 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.108s user 0.003s sys 0.011s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:30:49 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:30:49.606571 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:58537 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:30:49.818552 14761 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:30:49.818893 14761 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:30:49.819136 14761 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:30:49.829305 14761 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:30:49.829587 14761 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:30:49.829766 14761 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:30:49.829946 14761 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:30:49.830144 14761 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:30:49.830330 14761 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:30:49.851634 14761 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:58537 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:30:49.861563 14761 master_main.cc:80] Initializing master server... I0627 19:30:49.864503 14761 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 193592us Jun 27 19:30:49 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153049, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:30:49.869899 14761 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:30:49.874925 14761 fs_manager.cc:261] Metadata directory not provided I0627 19:30:49.875080 14761 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:30:49.875371 14761 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:30:49.875505 14761 server_base.cc:433] Attempting to create new FS layout instead I0627 19:30:49.884769 14761 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "a0c8a303760241bebbd50afc550235cb" format_stamp: "Formatted at 2018-06-28 02:30:49 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:49.885663 14761 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "a0c8a303760241bebbd50afc550235cb" format_stamp: "Formatted at 2018-06-28 02:30:49 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:49.897059 14761 fs_manager.cc:501] Time spent creating directory manager: real 0.011s user 0.007s sys 0.003s I0627 19:30:49.898298 14761 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:30:49.899626 14761 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:30:49.909759 14761 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s I0627 19:30:49.909909 14761 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/wal uuid: "a0c8a303760241bebbd50afc550235cb" format_stamp: "Formatted at 2018-06-28 02:30:49 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:49.910167 14761 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:30:50.197952 14761 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:30:50.201162 14761 master_main.cc:83] Starting Master server... I0627 19:30:50.315279 14761 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:58537 I0627 19:30:50.316292 14761 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:30:50.316417 14761 webserver.cc:186] Document root disabled I0627 19:30:50.318694 14761 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:58469/ I0627 19:30:50.319408 14761 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:30:50.324002 14877 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:30:50.326081 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 14761 WARNING: no policy specified for kudu/127.12.131.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.1@KRBTEST.COM" created. I0627 19:30:50.348832 14877 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb: Bootstrap starting. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. I0627 19:30:50.351608 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.12.131.1:0 --local_ip_for_outbound_sockets=127.12.131.1 --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:58537 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:30:50.351824 14877 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb: No blocks or log segments found. Creating new log. I0627 19:30:50.353981 14877 log.cc:525] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb: Log is configured to *not* fsync() on all Append() calls I0627 19:30:50.363416 14877 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb: No bootstrap required, opened a new log I0627 19:30:50.385259 14877 raft_consensus.cc:325] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c8a303760241bebbd50afc550235cb" member_type: VOTER } I0627 19:30:50.385833 14877 raft_consensus.cc:351] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:30:50.386060 14877 raft_consensus.cc:628] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c8a303760241bebbd50afc550235cb, State: Initialized, Role: FOLLOWER I0627 19:30:50.386818 14877 consensus_queue.cc:227] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c8a303760241bebbd50afc550235cb" member_type: VOTER } I0627 19:30:50.387356 14877 raft_consensus.cc:365] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:30:50.387617 14877 raft_consensus.cc:457] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:30:50.387856 14877 raft_consensus.cc:2730] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb: Snoozing failure detection for 2.116s (starting election) I0627 19:30:50.388118 14877 raft_consensus.cc:2782] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [term 0 FOLLOWER]: Advancing to term 1 I0627 19:30:50.392519 14877 raft_consensus.cc:479] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c8a303760241bebbd50afc550235cb" member_type: VOTER } I0627 19:30:50.393250 14877 leader_election.cc:270] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:30:50.398030 14882 raft_consensus.cc:2730] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb: Snoozing failure detection for 1.665s (election complete) I0627 19:30:50.398389 14882 raft_consensus.cc:2537] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:30:50.405690 14882 raft_consensus.cc:593] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [term 1 LEADER]: Becoming Leader. State: Replica: a0c8a303760241bebbd50afc550235cb, State: Running, Role: LEADER I0627 19:30:50.406250 14882 consensus_queue.cc:204] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c8a303760241bebbd50afc550235cb" member_type: VOTER } I0627 19:30:50.413506 14883 sys_catalog.cc:337] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0c8a303760241bebbd50afc550235cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c8a303760241bebbd50afc550235cb" member_type: VOTER } } I0627 19:30:50.414258 14883 sys_catalog.cc:340] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [sys.catalog]: This master's current role is: LEADER I0627 19:30:50.413679 14877 sys_catalog.cc:337] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "a0c8a303760241bebbd50afc550235cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c8a303760241bebbd50afc550235cb" member_type: VOTER } } I0627 19:30:50.513775 14877 sys_catalog.cc:340] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [sys.catalog]: This master's current role is: LEADER I0627 19:30:50.515350 14877 sys_catalog.cc:424] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [sys.catalog]: configured and running, proceeding with master startup. I0627 19:30:50.427721 14884 sys_catalog.cc:337] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0c8a303760241bebbd50afc550235cb. Latest consensus state: current_term: 1 leader_uuid: "a0c8a303760241bebbd50afc550235cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c8a303760241bebbd50afc550235cb" member_type: VOTER } } I0627 19:30:50.522575 14884 sys_catalog.cc:340] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb [sys.catalog]: This master's current role is: LEADER I0627 19:30:50.527318 14761 master_main.cc:86] Master server successfully started. W0627 19:30:50.531838 14887 catalog_manager.cc:1064] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:30:50.532109 14887 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:30:50.532443 14886 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:30:50.535760 14886 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:30:50.552204 14880 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:30:50.552742 14880 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.1 W0627 19:30:50.553174 14880 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:30:50.572283 14880 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:30:50.572453 14880 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:30:50.572645 14880 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.1 W0627 19:30:50.572968 14880 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.1 I0627 19:30:50.597321 14886 catalog_manager.cc:904] Generated new certificate authority record I0627 19:30:50.599601 14886 catalog_manager.cc:1026] Loading token signing keys... I0627 19:30:50.616271 14880 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.1:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:58537 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:30:50.619424 14886 catalog_manager.cc:4102] T 00000000000000000000000000000000 P a0c8a303760241bebbd50afc550235cb: Generated new TSK 0 I0627 19:30:50.631752 14880 tablet_server_main.cc:85] Initializing tablet server... I0627 19:30:50.634932 14880 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 194092us Jun 27 19:30:50 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153050, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:30:50.641460 14880 init.cc:339] Logged in from keytab as kudu/127.12.131.1@KRBTEST.COM (short username kudu) I0627 19:30:50.655977 14880 fs_manager.cc:261] Metadata directory not provided I0627 19:30:50.656226 14880 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:30:50.656641 14880 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:30:50.656852 14880 server_base.cc:433] Attempting to create new FS layout instead I0627 19:30:50.672217 14880 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data/instance: uuid: "fb74c9a7beb4407ba1ca75db9707d008" format_stamp: "Formatted at 2018-06-28 02:30:50 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:50.673295 14880 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/wal/instance: uuid: "fb74c9a7beb4407ba1ca75db9707d008" format_stamp: "Formatted at 2018-06-28 02:30:50 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:50.694594 14880 fs_manager.cc:501] Time spent creating directory manager: real 0.021s user 0.011s sys 0.008s I0627 19:30:50.696430 14880 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:30:50.697847 14880 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:30:50.714839 14880 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.001s sys 0.004s I0627 19:30:50.715087 14880 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/wal uuid: "fb74c9a7beb4407ba1ca75db9707d008" format_stamp: "Formatted at 2018-06-28 02:30:50 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:50.715484 14880 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:30:51.042371 14880 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:30:51.051096 14880 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:30:51.059545 14880 tablet_server_main.cc:90] Starting tablet server... I0627 19:30:51.683274 14880 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.1:57312 I0627 19:30:51.684581 14880 webserver.cc:175] Starting webserver on 127.12.131.1:0 I0627 19:30:51.684777 14880 webserver.cc:186] Document root disabled I0627 19:30:51.691475 14880 webserver.cc:313] Webserver started. Bound to: http://127.12.131.1:54822/ I0627 19:30:51.692572 14880 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data/info.pb I0627 19:30:51.697649 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 14880 I0627 19:30:51.723304 14880 tablet_server_main.cc:93] Tablet server successfully started. WARNING: no policy specified for kudu/127.12.131.2@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.2@KRBTEST.COM" created. Jun 27 19:30:51 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153050, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. I0627 19:30:51.767089 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.12.131.2:0 --local_ip_for_outbound_sockets=127.12.131.2 --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:58537 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:30:51.781502 15165 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:51.710095 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:40542 0627 19:30:51.729654 (+ 19559us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:51.729689 (+ 35us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:51.729759 (+ 70us) server_negotiation.cc:373] Connection header received 0627 19:30:51.742580 (+ 12821us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:51.742619 (+ 39us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:51.742853 (+ 234us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:51.743323 (+ 470us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:51.745938 (+ 2615us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:51.746607 (+ 669us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:51.748150 (+ 1543us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:51.749824 (+ 1674us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:51.750000 (+ 176us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:51.754857 (+ 4857us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:51.754949 (+ 92us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:51.754980 (+ 31us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:30:51.755066 (+ 86us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:51.762233 (+ 7167us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:51.765898 (+ 3665us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:51.765926 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:51.765946 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:51.766642 (+ 696us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:51.772964 (+ 6322us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:51.772994 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:51.773014 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:51.773683 (+ 669us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:51.774064 (+ 381us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:51.777707 (+ 3643us) server_negotiation.cc:264] Negotiation successful 0627 19:30:51.778134 (+ 427us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":19252,"thread_start_us":18122,"threads_started":1} I0627 19:30:51.783731 15163 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:51.708520 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:58537 0627 19:30:51.723907 (+ 15387us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:51.723980 (+ 73us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:51.741898 (+ 17918us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:51.743945 (+ 2047us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:51.743978 (+ 33us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:51.744756 (+ 778us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:30:51.745008 (+ 252us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:51.745684 (+ 676us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:51.745727 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:51.746882 (+ 1155us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:51.746906 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:51.747919 (+ 1013us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:51.747957 (+ 38us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:51.750493 (+ 2536us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:51.750517 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:51.750789 (+ 272us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:51.751054 (+ 265us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:30:51.751071 (+ 17us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:30:51.754077 (+ 3006us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:30:51.763004 (+ 8927us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:30:51.763032 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:30:51.763057 (+ 25us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:30:51.764320 (+ 1263us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:30:51.772042 (+ 7722us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:30:51.772070 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:30:51.772088 (+ 18us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:30:51.772344 (+ 256us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:30:51.774388 (+ 2044us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:30:51.774438 (+ 50us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:30:51.774893 (+ 455us) client_negotiation.cc:695] Sending connection context 0627 19:30:51.775431 (+ 538us) client_negotiation.cc:225] Negotiation successful 0627 19:30:51.775835 (+ 404us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":15064,"thread_start_us":11479,"threads_started":1} I0627 19:30:51.795622 15158 heartbeater.cc:346] Connected to a master server at 127.12.131.62:58537 I0627 19:30:51.795925 15158 heartbeater.cc:426] Registering TS with master... I0627 19:30:51.796507 15158 heartbeater.cc:475] Master 127.12.131.62:58537 requested a full tablet report, sending... I0627 19:30:51.798806 14842 ts_manager.cc:98] Registered new tserver with Master: fb74c9a7beb4407ba1ca75db9707d008 (127.12.131.1:57312) I0627 19:30:51.801784 14842 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.1@KRBTEST.COM'} at 127.12.131.1:40542 WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:30:52.167155 15175 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:30:52.169975 15175 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.2 W0627 19:30:52.170454 15175 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:30:52.207209 15175 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:30:52.207478 15175 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:30:52.207756 15175 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.2 W0627 19:30:52.208281 15175 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.2 I0627 19:30:52.273669 15175 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.2:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:58537 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.2 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:30:52.296116 15175 tablet_server_main.cc:85] Initializing tablet server... I0627 19:30:52.299895 15175 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 195092us Jun 27 19:30:52 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153052, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:30:52.322278 15175 init.cc:339] Logged in from keytab as kudu/127.12.131.2@KRBTEST.COM (short username kudu) I0627 19:30:52.341352 15175 fs_manager.cc:261] Metadata directory not provided I0627 19:30:52.341658 15175 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:30:52.342108 15175 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:30:52.342298 15175 server_base.cc:433] Attempting to create new FS layout instead I0627 19:30:52.357691 15175 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data/instance: uuid: "af678dd2fe7e4b94983b4cb1614af210" format_stamp: "Formatted at 2018-06-28 02:30:52 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:52.358985 15175 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/wal/instance: uuid: "af678dd2fe7e4b94983b4cb1614af210" format_stamp: "Formatted at 2018-06-28 02:30:52 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:52.389572 15175 fs_manager.cc:501] Time spent creating directory manager: real 0.030s user 0.012s sys 0.005s I0627 19:30:52.391450 15175 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:30:52.392067 15175 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:30:52.422150 15175 fs_manager.cc:417] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s I0627 19:30:52.422394 15175 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/wal uuid: "af678dd2fe7e4b94983b4cb1614af210" format_stamp: "Formatted at 2018-06-28 02:30:52 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:52.422751 15175 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:30:52.805202 15158 heartbeater.cc:467] Master 127.12.131.62:58537 was elected leader, sending a full tablet report... I0627 19:30:52.869580 15175 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:30:52.877974 15175 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:30:52.890971 15175 tablet_server_main.cc:90] Starting tablet server... I0627 19:30:53.496314 15175 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.2:55162 I0627 19:30:53.497627 15175 webserver.cc:175] Starting webserver on 127.12.131.2:0 I0627 19:30:53.497767 15175 webserver.cc:186] Document root disabled I0627 19:30:53.504657 15175 webserver.cc:313] Webserver started. Bound to: http://127.12.131.2:39880/ I0627 19:30:53.505501 15175 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data/info.pb I0627 19:30:53.510118 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 15175 I0627 19:30:53.525334 15175 tablet_server_main.cc:93] Tablet server successfully started. WARNING: no policy specified for kudu/127.12.131.3@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.3@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. I0627 19:30:53.550695 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.12.131.3:0 --local_ip_for_outbound_sockets=127.12.131.3 --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:58537 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:30:53.767968 15606 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:30:53.768497 15606 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.3 W0627 19:30:53.768908 15606 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:30:53.788205 15606 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:30:53.788461 15606 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:30:53.788769 15606 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.3 W0627 19:30:53.789305 15606 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.3 I0627 19:30:53.833170 15606 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.3:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:58537 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.3 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:30:53.860662 15606 tablet_server_main.cc:85] Initializing tablet server... I0627 19:30:53.863181 15606 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 195592us Jun 27 19:30:54 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153052, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM Jun 27 19:30:54 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153054, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:30:54.546716 15606 init.cc:339] Logged in from keytab as kudu/127.12.131.3@KRBTEST.COM (short username kudu) I0627 19:30:54.562367 15606 fs_manager.cc:261] Metadata directory not provided I0627 19:30:54.562628 15606 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:30:54.563024 15606 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:30:54.563230 15606 server_base.cc:433] Attempting to create new FS layout instead I0627 19:30:54.564010 15595 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:53.509272 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:58537 0627 19:30:53.517332 (+ 8060us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:53.517430 (+ 98us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:53.532151 (+ 14721us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:53.534216 (+ 2065us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:53.534248 (+ 32us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:53.535072 (+ 824us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:30:53.535362 (+ 290us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:53.535759 (+ 397us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:53.535807 (+ 48us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:53.539661 (+ 3854us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:53.539687 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:53.540767 (+ 1080us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:53.540813 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:53.542313 (+ 1500us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:53.542340 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:53.542660 (+ 320us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:53.542926 (+ 266us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:30:53.542947 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:30:54.546969 (+1004022us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:30:54.554918 (+ 7949us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:30:54.554937 (+ 19us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:30:54.554953 (+ 16us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:30:54.556022 (+ 1069us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:30:54.557864 (+ 1842us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:30:54.557892 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:30:54.557913 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:30:54.558160 (+ 247us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:30:54.560001 (+ 1841us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:30:54.560033 (+ 32us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:30:54.560472 (+ 439us) client_negotiation.cc:695] Sending connection context 0627 19:30:54.560950 (+ 478us) client_negotiation.cc:225] Negotiation successful 0627 19:30:54.561322 (+ 372us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":7587,"thread_start_us":7070,"threads_started":1} I0627 19:30:54.567451 15596 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:53.509791 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:36693 0627 19:30:53.519716 (+ 9925us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:53.519742 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:53.519782 (+ 40us) server_negotiation.cc:373] Connection header received 0627 19:30:53.532514 (+ 12732us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:53.532545 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:53.532671 (+ 126us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:53.532810 (+ 139us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:53.536006 (+ 3196us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:53.536599 (+ 593us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:53.540992 (+ 4393us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:53.542090 (+ 1098us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:53.542272 (+ 182us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:54.547630 (+1005358us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:54.547713 (+ 83us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:54.547739 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:30:54.547830 (+ 91us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:54.554508 (+ 6678us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:54.556598 (+ 2090us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:54.556626 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:54.556647 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:54.557291 (+ 644us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:54.558892 (+ 1601us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:54.558920 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:54.558940 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:54.559433 (+ 493us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:54.564611 (+ 5178us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:54.565035 (+ 424us) server_negotiation.cc:264] Negotiation successful 0627 19:30:54.565388 (+ 353us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":9715,"thread_start_us":8171,"threads_started":1} I0627 19:30:54.569674 15593 heartbeater.cc:346] Connected to a master server at 127.12.131.62:58537 I0627 19:30:54.569983 15593 heartbeater.cc:426] Registering TS with master... I0627 19:30:54.570627 15593 heartbeater.cc:475] Master 127.12.131.62:58537 requested a full tablet report, sending... I0627 19:30:54.572557 14842 ts_manager.cc:98] Registered new tserver with Master: af678dd2fe7e4b94983b4cb1614af210 (127.12.131.2:55162) I0627 19:30:54.574390 14842 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.2@KRBTEST.COM'} at 127.12.131.2:36693 I0627 19:30:54.579766 15606 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data/instance: uuid: "eaf05b48ba9646a0956ab0e4db5220c7" format_stamp: "Formatted at 2018-06-28 02:30:54 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:54.580799 15606 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/wal/instance: uuid: "eaf05b48ba9646a0956ab0e4db5220c7" format_stamp: "Formatted at 2018-06-28 02:30:54 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:54.598664 15606 fs_manager.cc:501] Time spent creating directory manager: real 0.017s user 0.013s sys 0.001s I0627 19:30:54.600481 15606 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:30:54.601094 15606 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:30:54.610560 15606 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0627 19:30:54.610812 15606 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/wal uuid: "eaf05b48ba9646a0956ab0e4db5220c7" format_stamp: "Formatted at 2018-06-28 02:30:54 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:54.611106 15606 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:30:54.802309 15606 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:30:54.806077 15606 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:30:54.807559 15606 tablet_server_main.cc:90] Starting tablet server... I0627 19:30:54.997817 15606 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.3:42055 I0627 19:30:54.998948 15606 webserver.cc:175] Starting webserver on 127.12.131.3:0 I0627 19:30:54.999094 15606 webserver.cc:186] Document root disabled I0627 19:30:55.000828 15606 webserver.cc:313] Webserver started. Bound to: http://127.12.131.3:58503/ I0627 19:30:55.001593 15606 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestNoKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data/info.pb I0627 19:30:55.007635 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 15606 I0627 19:30:55.012121 15606 tablet_server_main.cc:93] Tablet server successfully started. Jun 27 19:30:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153054, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM Jun 27 19:30:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[14752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:30:55.058225 15976 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:55.014733 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:58537 0627 19:30:55.020799 (+ 6066us) negotiation.cc:98] Waiting for socket to connect 0627 19:30:55.020864 (+ 65us) client_negotiation.cc:160] Beginning negotiation 0627 19:30:55.032001 (+ 11137us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:30:55.032837 (+ 836us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:30:55.032858 (+ 21us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:30:55.033521 (+ 663us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:30:55.033804 (+ 283us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:55.034185 (+ 381us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:55.034221 (+ 36us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:55.035071 (+ 850us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:55.035088 (+ 17us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:55.035887 (+ 799us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:30:55.035916 (+ 29us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:30:55.036995 (+ 1079us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:30:55.037026 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:30:55.037314 (+ 288us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:55.037731 (+ 417us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:30:55.037744 (+ 13us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:30:55.040436 (+ 2692us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:30:55.045905 (+ 5469us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:30:55.045925 (+ 20us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:30:55.045940 (+ 15us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:30:55.046882 (+ 942us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:30:55.052660 (+ 5778us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:30:55.052685 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:30:55.052704 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:30:55.052954 (+ 250us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:30:55.054360 (+ 1406us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:30:55.054413 (+ 53us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:30:55.054847 (+ 434us) client_negotiation.cc:695] Sending connection context 0627 19:30:55.055354 (+ 507us) client_negotiation.cc:225] Negotiation successful 0627 19:30:55.055756 (+ 402us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":5726,"thread_start_us":3876,"threads_started":1} I0627 19:30:55.058547 15977 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:55.016052 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:43697 0627 19:30:55.027224 (+ 11172us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:55.027248 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:55.027288 (+ 40us) server_negotiation.cc:373] Connection header received 0627 19:30:55.032289 (+ 5001us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:55.032317 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:55.032458 (+ 141us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:55.032599 (+ 141us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:55.034385 (+ 1786us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:55.034892 (+ 507us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:55.036083 (+ 1191us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:55.036779 (+ 696us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:55.036941 (+ 162us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:55.040881 (+ 3940us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:55.040932 (+ 51us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:55.040947 (+ 15us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:30:55.041000 (+ 53us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:55.045515 (+ 4515us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:55.047248 (+ 1733us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:55.047266 (+ 18us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:55.047277 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:55.052213 (+ 4936us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:55.053369 (+ 1156us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:55.053397 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:55.053451 (+ 54us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:55.053924 (+ 473us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:55.054346 (+ 422us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:55.055667 (+ 1321us) server_negotiation.cc:264] Negotiation successful 0627 19:30:55.056008 (+ 341us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":4036,"server-negotiator.queue_time_us":10956,"thread_start_us":11205,"threads_started":1} I0627 19:30:55.061044 15972 heartbeater.cc:346] Connected to a master server at 127.12.131.62:58537 I0627 19:30:55.061347 15972 heartbeater.cc:426] Registering TS with master... I0627 19:30:55.061991 15972 heartbeater.cc:475] Master 127.12.131.62:58537 requested a full tablet report, sending... I0627 19:30:55.063998 14842 ts_manager.cc:98] Registered new tserver with Master: eaf05b48ba9646a0956ab0e4db5220c7 (127.12.131.3:42055) I0627 19:30:55.065831 14842 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.3@KRBTEST.COM'} at 127.12.131.3:43697 I0627 19:30:55.071707 15596 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:55.009942 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:60224 0627 19:30:55.010259 (+ 317us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:55.010281 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:55.018694 (+ 8413us) server_negotiation.cc:373] Connection header received 0627 19:30:55.018985 (+ 291us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:55.019011 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:55.019144 (+ 133us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:55.019278 (+ 134us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:55.022534 (+ 3256us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:55.023019 (+ 485us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:55.033681 (+ 10662us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:55.034763 (+ 1082us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:55.034939 (+ 176us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:55.045897 (+ 10958us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:55.045928 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:55.045956 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:30:55.046045 (+ 89us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:55.055176 (+ 9131us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:55.061617 (+ 6441us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:55.061644 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:55.061662 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:55.064401 (+ 2739us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:55.065399 (+ 998us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:55.065454 (+ 55us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:55.065475 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:55.066862 (+ 1387us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:55.068968 (+ 2106us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:55.069176 (+ 208us) server_negotiation.cc:264] Negotiation successful 0627 19:30:55.069681 (+ 505us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":126} I0627 19:30:55.073729 12812 external_mini_cluster.cc:490] 3 TS(s) registered with all masters W0627 19:30:55.126603 15596 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:30:55.121837 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:60225 0627 19:30:55.122193 (+ 356us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:55.122215 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:55.123004 (+ 789us) server_negotiation.cc:373] Connection header received 0627 19:30:55.123266 (+ 262us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:55.123293 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:55.123452 (+ 159us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:55.123604 (+ 152us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:55.125759 (+ 2155us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.62:60225: BlockingRecv error: recv got EOF from 127.12.131.62:60225 (error 108) Metrics: {"server-negotiator.queue_time_us":143} I0627 19:30:55.141459 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 14880 W0627 19:30:55.195190 14779 connection.cc:512] server connection from 127.12.131.1:40542 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:40542): Connection reset by peer (error 104) I0627 19:30:55.196213 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 15175 W0627 19:30:55.225509 14780 connection.cc:512] server connection from 127.12.131.2:36693 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:36693): Connection reset by peer (error 104) I0627 19:30:55.226303 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 15606 W0627 19:30:55.259783 14801 connection.cc:512] server connection from 127.12.131.3:43697 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.3:43697): Connection reset by peer (error 104) I0627 19:30:55.260709 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 14761 [ OK ] SecurityITest.TestNoKerberosCredentials (5953 ms) [ RUN ] SecurityITest.SaslPlainFallback Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:30:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16093](info): setting up network... Jun 27 19:30:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16093](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:30:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16093](info): skipping unrecognized local address family 17 Jun 27 19:30:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16093](info): skipping unrecognized local address family 17 Jun 27 19:30:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16093](info): set up 1 sockets Jun 27 19:30:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16093](info): commencing operation krb5kdc: starting... W0627 19:30:55.554177 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.211s user 0.006s sys 0.012s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:30:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:30:55.748710 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:54890 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc-authentication=optional --user-acl=* --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:30:55.959707 16108 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:30:55.960285 16108 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:30:55.960681 16108 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:30:55.978410 16108 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:30:55.978644 16108 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:30:55.978809 16108 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:30:55.978978 16108 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:30:55.979182 16108 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:30:55.979420 16108 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:30:56.022836 16108 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:54890 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:30:56.038717 16108 master_main.cc:80] Initializing master server... I0627 19:30:56.042909 16108 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 197092us Jun 27 19:30:56 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153056, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:30:56.049741 16108 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:30:56.063482 16108 fs_manager.cc:261] Metadata directory not provided I0627 19:30:56.063725 16108 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:30:56.064381 16108 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:30:56.064611 16108 server_base.cc:433] Attempting to create new FS layout instead I0627 19:30:56.080562 16108 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "e61d34ad845748cb937d0f23211273c5" format_stamp: "Formatted at 2018-06-28 02:30:56 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:56.081702 16108 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "e61d34ad845748cb937d0f23211273c5" format_stamp: "Formatted at 2018-06-28 02:30:56 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:56.103363 16108 fs_manager.cc:501] Time spent creating directory manager: real 0.021s user 0.014s sys 0.004s I0627 19:30:56.105232 16108 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:30:56.106678 16108 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:30:56.130754 16108 fs_manager.cc:417] Time spent opening block manager: real 0.008s user 0.001s sys 0.005s I0627 19:30:56.131011 16108 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/wal uuid: "e61d34ad845748cb937d0f23211273c5" format_stamp: "Formatted at 2018-06-28 02:30:56 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:56.131386 16108 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:30:56.545831 16108 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:30:56.554215 16108 master_main.cc:83] Starting Master server... I0627 19:30:56.816174 16108 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:54890 I0627 19:30:56.817543 16108 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:30:56.817713 16108 webserver.cc:186] Document root disabled I0627 19:30:56.827229 16108 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:38608/ I0627 19:30:56.828310 16108 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.SaslPlainFallback.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:30:56.832260 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 16108 I0627 19:30:56.851047 16406 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed Jun 27 19:30:56 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:30:56.913594 16409 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:56.834661 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:55896 0627 19:30:56.848614 (+ 13953us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:56.848802 (+ 188us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:56.850648 (+ 1846us) server_negotiation.cc:373] Connection header received 0627 19:30:56.875741 (+ 25093us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:56.875807 (+ 66us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:56.876135 (+ 328us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:56.876560 (+ 425us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:56.880508 (+ 3948us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:56.884673 (+ 4165us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:56.886303 (+ 1630us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:56.888342 (+ 2039us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:56.888569 (+ 227us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:56.893359 (+ 4790us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:56.893394 (+ 35us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:56.893424 (+ 30us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:30:56.893542 (+ 118us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:56.900224 (+ 6682us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:56.902838 (+ 2614us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:56.902868 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:56.902893 (+ 25us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:56.903608 (+ 715us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:30:56.905181 (+ 1573us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:30:56.905207 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:30:56.905227 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:30:56.908685 (+ 3458us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:56.910766 (+ 2081us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:56.911063 (+ 297us) server_negotiation.cc:264] Negotiation successful 0627 19:30:56.911609 (+ 546us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5552,"thread_start_us":5130,"threads_started":1} I0627 19:30:56.916338 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:30:56.918565 16406 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5: Bootstrap starting. I0627 19:30:56.921674 16406 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5: No blocks or log segments found. Creating new log. I0627 19:30:56.924109 16406 log.cc:525] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5: Log is configured to *not* fsync() on all Append() calls I0627 19:30:56.931169 16406 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5: No bootstrap required, opened a new log I0627 19:30:56.954692 16406 raft_consensus.cc:325] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61d34ad845748cb937d0f23211273c5" member_type: VOTER } I0627 19:30:56.955955 16406 raft_consensus.cc:351] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:30:56.956226 16406 raft_consensus.cc:628] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61d34ad845748cb937d0f23211273c5, State: Initialized, Role: FOLLOWER I0627 19:30:56.957015 16406 consensus_queue.cc:227] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61d34ad845748cb937d0f23211273c5" member_type: VOTER } I0627 19:30:56.957367 16406 raft_consensus.cc:365] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:30:56.957623 16406 raft_consensus.cc:457] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:30:56.957792 16406 raft_consensus.cc:2730] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5: Snoozing failure detection for 1.620s (starting election) I0627 19:30:56.957965 16406 raft_consensus.cc:2782] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:30:56.960587 16406 raft_consensus.cc:479] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61d34ad845748cb937d0f23211273c5" member_type: VOTER } I0627 19:30:56.961166 16406 leader_election.cc:270] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:30:56.967411 16432 raft_consensus.cc:2730] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5: Snoozing failure detection for 2.152s (election complete) I0627 19:30:56.968699 16432 raft_consensus.cc:2537] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:30:56.974561 16432 raft_consensus.cc:593] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [term 1 LEADER]: Becoming Leader. State: Replica: e61d34ad845748cb937d0f23211273c5, State: Running, Role: LEADER I0627 19:30:56.975111 16432 consensus_queue.cc:204] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61d34ad845748cb937d0f23211273c5" member_type: VOTER } I0627 19:30:56.985692 16435 sys_catalog.cc:337] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e61d34ad845748cb937d0f23211273c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61d34ad845748cb937d0f23211273c5" member_type: VOTER } } I0627 19:30:56.986141 16435 sys_catalog.cc:340] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [sys.catalog]: This master's current role is: LEADER I0627 19:30:57.000931 16409 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:30:56.977407 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:55902 0627 19:30:56.977741 (+ 334us) server_negotiation.cc:176] Beginning negotiation 0627 19:30:56.977762 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:30:56.989819 (+ 12057us) server_negotiation.cc:373] Connection header received 0627 19:30:56.990223 (+ 404us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:30:56.990290 (+ 67us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:30:56.990470 (+ 180us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:30:56.990604 (+ 134us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:30:56.992944 (+ 2340us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:56.993316 (+ 372us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:56.994801 (+ 1485us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:30:56.995836 (+ 1035us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:30:56.996003 (+ 167us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:30:56.998695 (+ 2692us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:30:56.998721 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:30:56.998743 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:30:56.999202 (+ 459us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:30:56.999272 (+ 70us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:30:56.999324 (+ 52us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:30:56.999423 (+ 99us) server_negotiation.cc:901] Waiting for connection context 0627 19:30:56.999736 (+ 313us) server_negotiation.cc:264] Negotiation successful 0627 19:30:56.999922 (+ 186us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":120} I0627 19:30:56.985855 16436 sys_catalog.cc:337] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e61d34ad845748cb937d0f23211273c5. Latest consensus state: current_term: 1 leader_uuid: "e61d34ad845748cb937d0f23211273c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61d34ad845748cb937d0f23211273c5" member_type: VOTER } } I0627 19:30:57.324275 16436 sys_catalog.cc:340] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [sys.catalog]: This master's current role is: LEADER I0627 19:30:56.986527 16406 sys_catalog.cc:337] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "e61d34ad845748cb937d0f23211273c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61d34ad845748cb937d0f23211273c5" member_type: VOTER } } I0627 19:30:57.347681 16406 sys_catalog.cc:340] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [sys.catalog]: This master's current role is: LEADER I0627 19:30:57.348935 16406 sys_catalog.cc:424] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:30:57.370667 16108 master_main.cc:86] Master server successfully started. I0627 19:30:57.371177 16441 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:30:57.392319 16441 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... W0627 19:30:57.565554 16261 tablet.cc:1678] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:30:57.687399 16441 catalog_manager.cc:904] Generated new certificate authority record I0627 19:30:57.689723 16441 catalog_manager.cc:1026] Loading token signing keys... I0627 19:30:57.753433 16441 catalog_manager.cc:4102] T 00000000000000000000000000000000 P e61d34ad845748cb937d0f23211273c5: Generated new TSK 0 I0627 19:30:57.810192 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 16108 [ OK ] SecurityITest.SaslPlainFallback (2520 ms) [ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:30:57 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): setting up network... Jun 27 19:30:57 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:30:57 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): skipping unrecognized local address family 17 Jun 27 19:30:57 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): skipping unrecognized local address family 17 Jun 27 19:30:57 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): set up 1 sockets Jun 27 19:30:57 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): commencing operation krb5kdc: starting... W0627 19:30:58.043751 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.193s user 0.003s sys 0.015s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:30:58 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:30:58.278831 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:58110 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:30:58.560153 16725 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:30:58.560771 16725 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:30:58.561149 16725 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:30:58.578752 16725 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:30:58.579072 16725 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:30:58.579265 16725 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:30:58.580137 16725 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:30:58.580374 16725 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:30:58.590677 16725 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:30:58.654773 16725 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:58110 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:30:58.675823 16725 master_main.cc:80] Initializing master server... I0627 19:30:58.679615 16725 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 198092us Jun 27 19:30:58 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153058, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:30:58.686270 16725 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:30:58.689488 16725 fs_manager.cc:261] Metadata directory not provided I0627 19:30:58.689690 16725 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:30:58.690074 16725 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:30:58.690268 16725 server_base.cc:433] Attempting to create new FS layout instead I0627 19:30:58.699703 16725 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "655f1fcde62f407cb35a072fc26e4e92" format_stamp: "Formatted at 2018-06-28 02:30:58 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:58.700424 16725 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "655f1fcde62f407cb35a072fc26e4e92" format_stamp: "Formatted at 2018-06-28 02:30:58 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:58.711942 16725 fs_manager.cc:501] Time spent creating directory manager: real 0.011s user 0.008s sys 0.002s I0627 19:30:58.713263 16725 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:30:58.713757 16725 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:30:58.721688 16725 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0627 19:30:58.721832 16725 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/wal uuid: "655f1fcde62f407cb35a072fc26e4e92" format_stamp: "Formatted at 2018-06-28 02:30:58 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:58.722127 16725 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:30:58.925293 16725 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:30:58.927744 16725 master_main.cc:83] Starting Master server... I0627 19:30:59.042945 16725 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:58110 I0627 19:30:59.043895 16725 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:30:59.044008 16725 webserver.cc:186] Document root disabled I0627 19:30:59.046840 16725 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:42962/ I0627 19:30:59.047642 16725 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:30:59.050537 16925 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:30:59.053084 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 16725 WARNING: no policy specified for kudu/127.12.131.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.1@KRBTEST.COM" created. I0627 19:30:59.075887 16925 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92: Bootstrap starting. I0627 19:30:59.079156 16925 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92: No blocks or log segments found. Creating new log. I0627 19:30:59.080829 16925 log.cc:525] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92: Log is configured to *not* fsync() on all Append() calls I0627 19:30:59.087739 16925 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92: No bootstrap required, opened a new log Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. I0627 19:30:59.090020 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.12.131.1:0 --local_ip_for_outbound_sockets=127.12.131.1 --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:58110 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:30:59.108566 16925 raft_consensus.cc:325] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655f1fcde62f407cb35a072fc26e4e92" member_type: VOTER } I0627 19:30:59.109131 16925 raft_consensus.cc:351] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:30:59.109355 16925 raft_consensus.cc:628] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655f1fcde62f407cb35a072fc26e4e92, State: Initialized, Role: FOLLOWER I0627 19:30:59.110158 16925 consensus_queue.cc:227] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655f1fcde62f407cb35a072fc26e4e92" member_type: VOTER } I0627 19:30:59.110605 16925 raft_consensus.cc:365] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:30:59.110779 16925 raft_consensus.cc:457] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:30:59.110956 16925 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92: Snoozing failure detection for 1.863s (starting election) I0627 19:30:59.111140 16925 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:30:59.114262 16925 raft_consensus.cc:479] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655f1fcde62f407cb35a072fc26e4e92" member_type: VOTER } I0627 19:30:59.114748 16925 leader_election.cc:270] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:30:59.116549 16955 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92: Snoozing failure detection for 1.589s (election complete) I0627 19:30:59.117058 16955 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:30:59.123705 16955 raft_consensus.cc:593] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [term 1 LEADER]: Becoming Leader. State: Replica: 655f1fcde62f407cb35a072fc26e4e92, State: Running, Role: LEADER I0627 19:30:59.124195 16955 consensus_queue.cc:204] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655f1fcde62f407cb35a072fc26e4e92" member_type: VOTER } I0627 19:30:59.127934 16958 sys_catalog.cc:337] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 655f1fcde62f407cb35a072fc26e4e92. Latest consensus state: current_term: 1 leader_uuid: "655f1fcde62f407cb35a072fc26e4e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655f1fcde62f407cb35a072fc26e4e92" member_type: VOTER } } I0627 19:30:59.128469 16958 sys_catalog.cc:340] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [sys.catalog]: This master's current role is: LEADER I0627 19:30:59.128501 16925 sys_catalog.cc:337] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "655f1fcde62f407cb35a072fc26e4e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655f1fcde62f407cb35a072fc26e4e92" member_type: VOTER } } I0627 19:30:59.233481 16925 sys_catalog.cc:340] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [sys.catalog]: This master's current role is: LEADER I0627 19:30:59.234963 16925 sys_catalog.cc:424] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:30:59.129076 16957 sys_catalog.cc:337] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "655f1fcde62f407cb35a072fc26e4e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655f1fcde62f407cb35a072fc26e4e92" member_type: VOTER } } I0627 19:30:59.241026 16957 sys_catalog.cc:340] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92 [sys.catalog]: This master's current role is: LEADER I0627 19:30:59.245381 16961 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:30:59.250581 16725 master_main.cc:86] Master server successfully started. I0627 19:30:59.253171 16961 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0627 19:30:59.315047 16961 catalog_manager.cc:904] Generated new certificate authority record I0627 19:30:59.322648 16961 catalog_manager.cc:1026] Loading token signing keys... I0627 19:30:59.385159 16961 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 655f1fcde62f407cb35a072fc26e4e92: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:30:59.424468 16947 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:30:59.424921 16947 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.1 W0627 19:30:59.426002 16947 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:30:59.445924 16947 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:30:59.446077 16947 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:30:59.446285 16947 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.1 W0627 19:30:59.446652 16947 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.1 I0627 19:30:59.476048 16947 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.1:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:58110 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:30:59.484997 16947 tablet_server_main.cc:85] Initializing tablet server... I0627 19:30:59.486814 16947 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 198592us Jun 27 19:30:59 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153059, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:30:59.489141 16947 init.cc:339] Logged in from keytab as kudu/127.12.131.1@KRBTEST.COM (short username kudu) I0627 19:30:59.493419 16947 fs_manager.cc:261] Metadata directory not provided I0627 19:30:59.493590 16947 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:30:59.493962 16947 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:30:59.494139 16947 server_base.cc:433] Attempting to create new FS layout instead I0627 19:30:59.530443 16947 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data/instance: uuid: "a2065c071ec84dcbb4410c6366458113" format_stamp: "Formatted at 2018-06-28 02:30:59 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:59.547960 16947 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/wal/instance: uuid: "a2065c071ec84dcbb4410c6366458113" format_stamp: "Formatted at 2018-06-28 02:30:59 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:59.570849 16947 fs_manager.cc:501] Time spent creating directory manager: real 0.021s user 0.008s sys 0.008s I0627 19:30:59.572834 16947 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:30:59.573537 16947 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:30:59.594825 16947 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.002s sys 0.004s I0627 19:30:59.595067 16947 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/wal uuid: "a2065c071ec84dcbb4410c6366458113" format_stamp: "Formatted at 2018-06-28 02:30:59 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:30:59.595475 16947 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:30:59.835263 16947 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:30:59.841701 16947 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:30:59.844821 16947 tablet_server_main.cc:90] Starting tablet server... I0627 19:31:00.470830 16947 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.1:46356 I0627 19:31:00.472637 16947 webserver.cc:175] Starting webserver on 127.12.131.1:0 I0627 19:31:00.472828 16947 webserver.cc:186] Document root disabled I0627 19:31:00.474833 16947 webserver.cc:313] Webserver started. Bound to: http://127.12.131.1:53529/ I0627 19:31:00.475883 16947 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-0/data/info.pb I0627 19:31:00.480777 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 16947 I0627 19:31:00.495465 16947 tablet_server_main.cc:93] Tablet server successfully started. WARNING: no policy specified for kudu/127.12.131.2@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.2@KRBTEST.COM" created. Jun 27 19:31:00 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153059, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. I0627 19:31:00.521986 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.12.131.2:0 --local_ip_for_outbound_sockets=127.12.131.2 --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:58110 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:31:00.539750 17187 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:00.484517 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:58110 0627 19:31:00.492935 (+ 8418us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:00.493044 (+ 109us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:00.508501 (+ 15457us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:00.510152 (+ 1651us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:00.510186 (+ 34us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:00.511158 (+ 972us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:00.511477 (+ 319us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:00.511901 (+ 424us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:00.511948 (+ 47us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:00.513152 (+ 1204us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:00.513179 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:00.514341 (+ 1162us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:00.514415 (+ 74us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:00.517278 (+ 2863us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:00.517307 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:00.517642 (+ 335us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:00.517932 (+ 290us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:00.517953 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:00.521185 (+ 3232us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:00.530288 (+ 9103us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:00.530320 (+ 32us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:00.530347 (+ 27us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:00.531806 (+ 1459us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:00.533713 (+ 1907us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:00.533740 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:00.533761 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:00.534011 (+ 250us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:00.535921 (+ 1910us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:00.535951 (+ 30us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:00.536425 (+ 474us) client_negotiation.cc:695] Sending connection context 0627 19:31:00.536921 (+ 496us) client_negotiation.cc:225] Negotiation successful 0627 19:31:00.537319 (+ 398us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":7827,"thread_start_us":6015,"threads_started":1} I0627 19:31:00.543300 17189 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:00.486107 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:49439 0627 19:31:00.497701 (+ 11594us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:00.497740 (+ 39us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:00.497813 (+ 73us) server_negotiation.cc:373] Connection header received 0627 19:31:00.509304 (+ 11491us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:00.509345 (+ 41us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:00.509636 (+ 291us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:00.509978 (+ 342us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:00.512148 (+ 2170us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:00.512937 (+ 789us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:00.514637 (+ 1700us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:00.516479 (+ 1842us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:00.516678 (+ 199us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:00.521917 (+ 5239us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:00.522007 (+ 90us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:00.522038 (+ 31us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:00.522120 (+ 82us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:00.529523 (+ 7403us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:00.532417 (+ 2894us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:00.532449 (+ 32us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:00.532473 (+ 24us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:00.533150 (+ 677us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:00.534563 (+ 1413us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:00.534591 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:00.534609 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:00.535205 (+ 596us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:00.535579 (+ 374us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:00.540735 (+ 5156us) server_negotiation.cc:264] Negotiation successful 0627 19:31:00.541162 (+ 427us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":11179,"thread_start_us":5283,"threads_started":1} I0627 19:31:00.546468 17185 heartbeater.cc:346] Connected to a master server at 127.12.131.62:58110 I0627 19:31:00.546802 17185 heartbeater.cc:426] Registering TS with master... I0627 19:31:00.547454 17185 heartbeater.cc:475] Master 127.12.131.62:58110 requested a full tablet report, sending... I0627 19:31:00.549788 16869 ts_manager.cc:98] Registered new tserver with Master: a2065c071ec84dcbb4410c6366458113 (127.12.131.1:46356) I0627 19:31:00.552815 16869 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.1@KRBTEST.COM'} at 127.12.131.1:49439 WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:00.795753 17191 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:00.796249 17191 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.2 W0627 19:31:00.796712 17191 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:00.820205 17191 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:00.820487 17191 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:00.820775 17191 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.2 W0627 19:31:00.821293 17191 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.2 I0627 19:31:00.875291 17191 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.2:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:58110 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.2 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:00.905305 17191 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:00.909587 17191 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 199092us Jun 27 19:31:00 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153060, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:00.919365 17191 init.cc:339] Logged in from keytab as kudu/127.12.131.2@KRBTEST.COM (short username kudu) I0627 19:31:00.925535 17191 fs_manager.cc:261] Metadata directory not provided I0627 19:31:00.925777 17191 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:00.926282 17191 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:00.926509 17191 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:00.943881 17191 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data/instance: uuid: "e712d9fea01342c38371c1480e1aa420" format_stamp: "Formatted at 2018-06-28 02:31:00 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:00.945058 17191 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/wal/instance: uuid: "e712d9fea01342c38371c1480e1aa420" format_stamp: "Formatted at 2018-06-28 02:31:00 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:00.975136 17191 fs_manager.cc:501] Time spent creating directory manager: real 0.030s user 0.012s sys 0.002s I0627 19:31:00.977001 17191 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:00.979802 17191 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:01.004783 17191 fs_manager.cc:417] Time spent opening block manager: real 0.014s user 0.001s sys 0.009s I0627 19:31:01.005030 17191 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/wal uuid: "e712d9fea01342c38371c1480e1aa420" format_stamp: "Formatted at 2018-06-28 02:31:00 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:01.005908 17191 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:01.315075 17191 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:01.319562 17191 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:01.321665 17191 tablet_server_main.cc:90] Starting tablet server... I0627 19:31:01.556422 17185 heartbeater.cc:467] Master 127.12.131.62:58110 was elected leader, sending a full tablet report... I0627 19:31:01.796248 17191 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.2:34660 I0627 19:31:01.797302 17191 webserver.cc:175] Starting webserver on 127.12.131.2:0 I0627 19:31:01.797437 17191 webserver.cc:186] Document root disabled I0627 19:31:01.800052 17191 webserver.cc:313] Webserver started. Bound to: http://127.12.131.2:42410/ I0627 19:31:01.800765 17191 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-1/data/info.pb I0627 19:31:01.806463 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 17191 I0627 19:31:01.812098 17191 tablet_server_main.cc:93] Tablet server successfully started. WARNING: no policy specified for kudu/127.12.131.3@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.3@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. I0627 19:31:01.833317 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.12.131.3:0 --local_ip_for_outbound_sockets=127.12.131.3 --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:58110 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} Jun 27 19:31:01 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153060, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:01.852825 17411 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:01.806763 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:52776 0627 19:31:01.808523 (+ 1760us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:01.808549 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:01.810854 (+ 2305us) server_negotiation.cc:373] Connection header received 0627 19:31:01.824844 (+ 13990us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:01.824876 (+ 32us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:01.825007 (+ 131us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:01.825151 (+ 144us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:01.827747 (+ 2596us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:01.828232 (+ 485us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:01.829822 (+ 1590us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:01.830867 (+ 1045us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:01.831034 (+ 167us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:01.835565 (+ 4531us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:01.835635 (+ 70us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:01.835658 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:01.835732 (+ 74us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:01.841336 (+ 5604us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:01.843888 (+ 2552us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:01.843916 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:01.843932 (+ 16us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:01.844613 (+ 681us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:01.846056 (+ 1443us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:01.846080 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:01.846097 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:01.846556 (+ 459us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:01.846869 (+ 313us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:01.850820 (+ 3951us) server_negotiation.cc:264] Negotiation successful 0627 19:31:01.851155 (+ 335us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1525,"thread_start_us":1271,"threads_started":1} I0627 19:31:01.854346 17412 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:01.807066 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:58110 0627 19:31:01.810572 (+ 3506us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:01.810683 (+ 111us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:01.824436 (+ 13753us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:01.825704 (+ 1268us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:01.825736 (+ 32us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:01.826529 (+ 793us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:01.826800 (+ 271us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:01.827494 (+ 694us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:01.827536 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:01.828520 (+ 984us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:01.828544 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:01.829572 (+ 1028us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:01.829610 (+ 38us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:01.831508 (+ 1898us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:01.831534 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:01.831826 (+ 292us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:01.832073 (+ 247us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:01.832093 (+ 20us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:01.834883 (+ 2790us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:01.842029 (+ 7146us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:01.842058 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:01.842083 (+ 25us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:01.843334 (+ 1251us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:01.845244 (+ 1910us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:01.845269 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:01.845289 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:01.845535 (+ 246us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:01.847192 (+ 1657us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:01.847220 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:01.847651 (+ 431us) client_negotiation.cc:695] Sending connection context 0627 19:31:01.848099 (+ 448us) client_negotiation.cc:225] Negotiation successful 0627 19:31:01.848504 (+ 405us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":2986,"thread_start_us":2377,"threads_started":1} I0627 19:31:01.856753 17409 heartbeater.cc:346] Connected to a master server at 127.12.131.62:58110 I0627 19:31:01.857054 17409 heartbeater.cc:426] Registering TS with master... I0627 19:31:01.857663 17409 heartbeater.cc:475] Master 127.12.131.62:58110 requested a full tablet report, sending... I0627 19:31:01.859393 16869 ts_manager.cc:98] Registered new tserver with Master: e712d9fea01342c38371c1480e1aa420 (127.12.131.2:34660) I0627 19:31:01.860538 16869 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.2@KRBTEST.COM'} at 127.12.131.2:52776 WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:01.999017 17417 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:01.999320 17417 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.3 W0627 19:31:01.999590 17417 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:02.008168 17417 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:02.008304 17417 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:02.008499 17417 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.3 W0627 19:31:02.008807 17417 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.3 I0627 19:31:02.031486 17417 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.3:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:58110 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.3 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:02.041374 17417 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:02.044260 17417 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 200092us Jun 27 19:31:02 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153062, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:02.049569 17417 init.cc:339] Logged in from keytab as kudu/127.12.131.3@KRBTEST.COM (short username kudu) I0627 19:31:02.054738 17417 fs_manager.cc:261] Metadata directory not provided I0627 19:31:02.054884 17417 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:02.055141 17417 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:02.055268 17417 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:02.067629 17417 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data/instance: uuid: "e1337da5be4d496888cf8a59ce9a86c0" format_stamp: "Formatted at 2018-06-28 02:31:02 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:02.068352 17417 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/wal/instance: uuid: "e1337da5be4d496888cf8a59ce9a86c0" format_stamp: "Formatted at 2018-06-28 02:31:02 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:02.080617 17417 fs_manager.cc:501] Time spent creating directory manager: real 0.012s user 0.008s sys 0.004s I0627 19:31:02.081815 17417 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:02.083012 17417 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:02.096263 17417 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.002s sys 0.004s I0627 19:31:02.096431 17417 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/wal uuid: "e1337da5be4d496888cf8a59ce9a86c0" format_stamp: "Formatted at 2018-06-28 02:31:02 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:02.096676 17417 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:02.330263 17417 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:02.338745 17417 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:02.341042 17417 tablet_server_main.cc:90] Starting tablet server... I0627 19:31:02.637008 17417 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.3:40797 I0627 19:31:02.638437 17417 webserver.cc:175] Starting webserver on 127.12.131.3:0 I0627 19:31:02.638564 17417 webserver.cc:186] Document root disabled I0627 19:31:02.641216 17417 webserver.cc:313] Webserver started. Bound to: http://127.12.131.3:60376/ I0627 19:31:02.641921 17417 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1530153042096250-12812/minicluster-data/ts-2/data/info.pb I0627 19:31:02.642920 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 17417 I0627 19:31:02.659353 17417 tablet_server_main.cc:93] Tablet server successfully started. Jun 27 19:31:02 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM Jun 27 19:31:02 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153062, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:02.685169 17611 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:02.645031 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:58370 0627 19:31:02.654431 (+ 9400us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:02.654459 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:02.654498 (+ 39us) server_negotiation.cc:373] Connection header received 0627 19:31:02.654609 (+ 111us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:02.654634 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:02.654772 (+ 138us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:02.658672 (+ 3900us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:02.658936 (+ 264us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:02.659533 (+ 597us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:02.661233 (+ 1700us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:02.662275 (+ 1042us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:02.662466 (+ 191us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:02.668670 (+ 6204us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:02.668699 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:02.668720 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:02.668805 (+ 85us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:02.674612 (+ 5807us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:02.676332 (+ 1720us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:02.676358 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:02.676376 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:02.677091 (+ 715us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:02.677707 (+ 616us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:02.677730 (+ 23us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:02.677746 (+ 16us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:02.680718 (+ 2972us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:02.681096 (+ 378us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:02.682983 (+ 1887us) server_negotiation.cc:264] Negotiation successful 0627 19:31:02.683452 (+ 469us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":2477,"server-negotiator.queue_time_us":9107,"thread_start_us":17513,"threads_started":1} I0627 19:31:02.693852 17608 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:02.651475 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:50556 0627 19:31:02.662506 (+ 11031us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:02.662534 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:02.662572 (+ 38us) server_negotiation.cc:373] Connection header received 0627 19:31:02.669474 (+ 6902us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:02.669503 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:02.669596 (+ 93us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:02.669741 (+ 145us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:02.671509 (+ 1768us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:02.671914 (+ 405us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:02.672923 (+ 1009us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:02.673599 (+ 676us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:02.673762 (+ 163us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:02.676615 (+ 2853us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:02.676669 (+ 54us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:02.676683 (+ 14us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:02.676740 (+ 57us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:02.681331 (+ 4591us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:02.684207 (+ 2876us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:02.684236 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:02.684255 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:02.684921 (+ 666us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:02.686507 (+ 1586us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:02.686533 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:02.686552 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:02.688036 (+ 1484us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:02.688392 (+ 356us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:02.689430 (+ 1038us) server_negotiation.cc:264] Negotiation successful 0627 19:31:02.690609 (+ 1179us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":194,"server-negotiator.queue_time_us":10721,"thread_start_us":2397,"threads_started":1} I0627 19:31:02.695067 17612 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:02.652136 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:58110 0627 19:31:02.659536 (+ 7400us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:02.659613 (+ 77us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:02.669190 (+ 9577us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:02.669970 (+ 780us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:02.669992 (+ 22us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:02.670538 (+ 546us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:02.670738 (+ 200us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:02.671317 (+ 579us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:02.671344 (+ 27us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:02.672073 (+ 729us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:02.672088 (+ 15us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:02.672755 (+ 667us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:02.672781 (+ 26us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:02.673795 (+ 1014us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:02.673820 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:02.674089 (+ 269us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:02.674250 (+ 161us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:02.674262 (+ 12us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:02.676182 (+ 1920us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:02.681968 (+ 5786us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:02.681997 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:02.682023 (+ 26us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:02.683641 (+ 1618us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:02.685624 (+ 1983us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:02.685648 (+ 24us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:02.685665 (+ 17us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:02.685914 (+ 249us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:02.688488 (+ 2574us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:02.688506 (+ 18us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:02.688798 (+ 292us) client_negotiation.cc:695] Sending connection context 0627 19:31:02.689163 (+ 365us) client_negotiation.cc:225] Negotiation successful 0627 19:31:02.690236 (+ 1073us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":6949,"thread_start_us":6310,"threads_started":1} I0627 19:31:02.697768 17606 heartbeater.cc:346] Connected to a master server at 127.12.131.62:58110 I0627 19:31:02.698009 17606 heartbeater.cc:426] Registering TS with master... I0627 19:31:02.698529 17606 heartbeater.cc:475] Master 127.12.131.62:58110 requested a full tablet report, sending... I0627 19:31:02.700671 16867 ts_manager.cc:98] Registered new tserver with Master: e1337da5be4d496888cf8a59ce9a86c0 (127.12.131.3:40797) I0627 19:31:02.702544 16867 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.3@KRBTEST.COM'} at 127.12.131.3:50556 I0627 19:31:02.703410 12812 external_mini_cluster.cc:490] 3 TS(s) registered with all masters Jun 27 19:31:02 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153062, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for joe-interloper@KRBTEST.COM: Jun 27 19:31:02 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[16650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153062, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:02.789764 17608 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:02.754937 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:58371 0627 19:31:02.755227 (+ 290us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:02.755251 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:02.764142 (+ 8891us) server_negotiation.cc:373] Connection header received 0627 19:31:02.764593 (+ 451us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:02.764621 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:02.764751 (+ 130us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:02.764881 (+ 130us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:02.767235 (+ 2354us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:02.768073 (+ 838us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:02.769690 (+ 1617us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:02.770718 (+ 1028us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:02.770874 (+ 156us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:02.775563 (+ 4689us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:02.775589 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:02.775610 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:02.775686 (+ 76us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:02.781745 (+ 6059us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:02.783597 (+ 1852us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:02.783623 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:02.783640 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:02.784249 (+ 609us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:02.785142 (+ 893us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:02.785167 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:02.785185 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:02.785662 (+ 477us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:02.785946 (+ 284us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:02.787714 (+ 1768us) server_negotiation.cc:264] Negotiation successful 0627 19:31:02.788136 (+ 422us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":93} W0627 19:31:02.790772 16867 server_base.cc:564] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.12.131.62:58371 I0627 19:31:02.795382 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 16947 W0627 19:31:02.821120 16793 connection.cc:512] server connection from 127.12.131.1:49439 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:49439): Connection reset by peer (error 104) I0627 19:31:02.822870 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 17191 W0627 19:31:02.858047 16793 connection.cc:512] server connection from 127.12.131.2:52776 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:52776): Connection reset by peer (error 104) I0627 19:31:02.858083 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 17417 W0627 19:31:02.887554 16793 connection.cc:512] server connection from 127.12.131.3:50556 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.3:50556): Connection reset by peer (error 104) I0627 19:31:02.887554 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 16725 [ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5083 ms) [ RUN ] SecurityITest.TestAuthorizedSuperuser Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): setting up network... Jun 27 19:31:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): skipping unrecognized local address family 17 Jun 27 19:31:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): skipping unrecognized local address family 17 Jun 27 19:31:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): set up 1 sockets Jun 27 19:31:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): commencing operation krb5kdc: starting... W0627 19:31:03.077919 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.149s user 0.004s sys 0.012s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:31:03.222836 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:45576 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:03.422916 17774 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:03.423213 17774 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:31:03.423430 17774 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:03.433575 17774 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:03.433820 17774 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:03.434000 17774 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:03.434171 17774 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:03.434367 17774 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:03.434612 17774 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:31:03.460422 17774 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:45576 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:03.469040 17774 master_main.cc:80] Initializing master server... I0627 19:31:03.472578 17774 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 200592us Jun 27 19:31:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153063, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:03.478979 17774 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:31:03.485479 17774 fs_manager.cc:261] Metadata directory not provided I0627 19:31:03.485628 17774 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:03.485891 17774 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:03.486013 17774 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:03.501690 17774 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "e437ce93c7084c75bd8ccc783e312ce9" format_stamp: "Formatted at 2018-06-28 02:31:03 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:03.502792 17774 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "e437ce93c7084c75bd8ccc783e312ce9" format_stamp: "Formatted at 2018-06-28 02:31:03 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:03.524806 17774 fs_manager.cc:501] Time spent creating directory manager: real 0.022s user 0.012s sys 0.007s I0627 19:31:03.526690 17774 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:03.528134 17774 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:03.548331 17774 fs_manager.cc:417] Time spent opening block manager: real 0.008s user 0.002s sys 0.004s I0627 19:31:03.548599 17774 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/wal uuid: "e437ce93c7084c75bd8ccc783e312ce9" format_stamp: "Formatted at 2018-06-28 02:31:03 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:03.548954 17774 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:03.902976 17774 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:03.909984 17774 master_main.cc:83] Starting Master server... I0627 19:31:04.172513 17774 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:45576 I0627 19:31:04.173709 17774 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:31:04.173820 17774 webserver.cc:186] Document root disabled I0627 19:31:04.176074 17774 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:36688/ I0627 19:31:04.176856 17774 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:04.181205 17975 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:04.183137 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 17774 I0627 19:31:04.206748 17975 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9: Bootstrap starting. I0627 19:31:04.210067 17975 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9: No blocks or log segments found. Creating new log. WARNING: no policy specified for kudu/127.12.131.1@KRBTEST.COM; defaulting to no policy I0627 19:31:04.213634 17975 log.cc:525] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9: Log is configured to *not* fsync() on all Append() calls Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.1@KRBTEST.COM" created. I0627 19:31:04.225219 17975 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9: No bootstrap required, opened a new log Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. I0627 19:31:04.248486 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.12.131.1:0 --local_ip_for_outbound_sockets=127.12.131.1 --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:45576 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:31:04.251544 17975 raft_consensus.cc:325] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e437ce93c7084c75bd8ccc783e312ce9" member_type: VOTER } I0627 19:31:04.252022 17975 raft_consensus.cc:351] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:04.252213 17975 raft_consensus.cc:628] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e437ce93c7084c75bd8ccc783e312ce9, State: Initialized, Role: FOLLOWER I0627 19:31:04.253104 17975 consensus_queue.cc:227] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e437ce93c7084c75bd8ccc783e312ce9" member_type: VOTER } I0627 19:31:04.253664 17975 raft_consensus.cc:365] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:04.253935 17975 raft_consensus.cc:457] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:04.254211 17975 raft_consensus.cc:2730] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9: Snoozing failure detection for 1.963s (starting election) I0627 19:31:04.254561 17975 raft_consensus.cc:2782] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:04.260061 17975 raft_consensus.cc:479] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e437ce93c7084c75bd8ccc783e312ce9" member_type: VOTER } I0627 19:31:04.260670 17975 leader_election.cc:270] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:04.265760 17989 raft_consensus.cc:2730] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9: Snoozing failure detection for 2.176s (election complete) I0627 19:31:04.270069 17989 raft_consensus.cc:2537] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:04.285255 17989 raft_consensus.cc:593] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [term 1 LEADER]: Becoming Leader. State: Replica: e437ce93c7084c75bd8ccc783e312ce9, State: Running, Role: LEADER I0627 19:31:04.285920 17989 consensus_queue.cc:204] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e437ce93c7084c75bd8ccc783e312ce9" member_type: VOTER } I0627 19:31:04.298254 17991 sys_catalog.cc:337] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e437ce93c7084c75bd8ccc783e312ce9. Latest consensus state: current_term: 1 leader_uuid: "e437ce93c7084c75bd8ccc783e312ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e437ce93c7084c75bd8ccc783e312ce9" member_type: VOTER } } I0627 19:31:04.298769 17991 sys_catalog.cc:340] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [sys.catalog]: This master's current role is: LEADER I0627 19:31:04.300806 17990 sys_catalog.cc:337] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e437ce93c7084c75bd8ccc783e312ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e437ce93c7084c75bd8ccc783e312ce9" member_type: VOTER } } I0627 19:31:04.667662 17990 sys_catalog.cc:340] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [sys.catalog]: This master's current role is: LEADER I0627 19:31:04.298892 17975 sys_catalog.cc:337] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "e437ce93c7084c75bd8ccc783e312ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e437ce93c7084c75bd8ccc783e312ce9" member_type: VOTER } } I0627 19:31:04.675727 17975 sys_catalog.cc:340] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [sys.catalog]: This master's current role is: LEADER I0627 19:31:04.677047 17975 sys_catalog.cc:424] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:04.689841 17774 master_main.cc:86] Master server successfully started. W0627 19:31:04.695801 18007 catalog_manager.cc:1064] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:04.696111 18007 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:31:04.696372 17994 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:04.699558 17994 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0627 19:31:04.856564 17994 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:04.858816 17994 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:04.891427 17994 catalog_manager.cc:4102] T 00000000000000000000000000000000 P e437ce93c7084c75bd8ccc783e312ce9: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:04.901638 17987 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:04.903877 17987 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.1 W0627 19:31:04.908618 17987 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:04.947070 17987 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:04.949206 17987 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:04.949529 17987 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.1 W0627 19:31:04.950045 17987 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.1 I0627 19:31:05.016979 17987 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.1:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:45576 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:05.032166 17987 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:05.035800 17987 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 201592us Jun 27 19:31:05 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153065, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:05.044916 17987 init.cc:339] Logged in from keytab as kudu/127.12.131.1@KRBTEST.COM (short username kudu) I0627 19:31:05.074870 17987 fs_manager.cc:261] Metadata directory not provided I0627 19:31:05.075139 17987 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:05.075570 17987 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:05.075778 17987 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:05.104576 17987 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/data/instance: uuid: "af2316a3e9f04cab81bf9d17c109f63f" format_stamp: "Formatted at 2018-06-28 02:31:05 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:05.106389 17987 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/wal/instance: uuid: "af2316a3e9f04cab81bf9d17c109f63f" format_stamp: "Formatted at 2018-06-28 02:31:05 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:05.157491 17987 fs_manager.cc:501] Time spent creating directory manager: real 0.050s user 0.013s sys 0.007s I0627 19:31:05.166441 17987 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:05.167894 17987 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:05.211674 17987 fs_manager.cc:417] Time spent opening block manager: real 0.020s user 0.001s sys 0.004s I0627 19:31:05.211917 17987 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/wal uuid: "af2316a3e9f04cab81bf9d17c109f63f" format_stamp: "Formatted at 2018-06-28 02:31:05 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:05.212296 17987 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:05.838122 17987 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:05.848392 17987 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:05.854868 17987 tablet_server_main.cc:90] Starting tablet server... I0627 19:31:06.705133 17987 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.1:33066 I0627 19:31:06.706583 17987 webserver.cc:175] Starting webserver on 127.12.131.1:0 I0627 19:31:06.706763 17987 webserver.cc:186] Document root disabled I0627 19:31:06.712813 17987 webserver.cc:313] Webserver started. Bound to: http://127.12.131.1:53516/ I0627 19:31:06.713827 17987 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-0/data/info.pb I0627 19:31:06.718655 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 17987 I0627 19:31:06.733798 17987 tablet_server_main.cc:93] Tablet server successfully started. WARNING: no policy specified for kudu/127.12.131.2@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.2@KRBTEST.COM" created. Jun 27 19:31:06 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153065, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:06.779204 18297 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:06.729498 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:45576 0627 19:31:06.735001 (+ 5503us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:06.735125 (+ 124us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:06.749684 (+ 14559us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:06.751025 (+ 1341us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:06.751058 (+ 33us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:06.751890 (+ 832us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:06.752164 (+ 274us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:06.752568 (+ 404us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:06.752617 (+ 49us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:06.753483 (+ 866us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:06.753510 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:06.754632 (+ 1122us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:06.754677 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:06.756421 (+ 1744us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:06.756451 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:06.756759 (+ 308us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:06.757018 (+ 259us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:06.757040 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:06.760381 (+ 3341us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:06.766543 (+ 6162us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:06.766574 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:06.766600 (+ 26us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:06.767914 (+ 1314us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:06.769207 (+ 1293us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:06.769232 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:06.769252 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:06.769496 (+ 244us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:06.775754 (+ 6258us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:06.775777 (+ 23us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:06.776041 (+ 264us) client_negotiation.cc:695] Sending connection context 0627 19:31:06.776422 (+ 381us) client_negotiation.cc:225] Negotiation successful 0627 19:31:06.776794 (+ 372us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":5080,"thread_start_us":4632,"threads_started":1} I0627 19:31:06.779528 18296 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:06.729271 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:53730 0627 19:31:06.737311 (+ 8040us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:06.737347 (+ 36us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:06.737448 (+ 101us) server_negotiation.cc:373] Connection header received 0627 19:31:06.750327 (+ 12879us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:06.750360 (+ 33us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:06.750568 (+ 208us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:06.750848 (+ 280us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:06.752801 (+ 1953us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:06.753278 (+ 477us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:06.754845 (+ 1567us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:06.756183 (+ 1338us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:06.756377 (+ 194us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:06.760898 (+ 4521us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:06.760951 (+ 53us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:06.760972 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:06.761029 (+ 57us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:06.766016 (+ 4987us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:06.768339 (+ 2323us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:06.768368 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:06.768389 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:06.768847 (+ 458us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:06.774633 (+ 5786us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:06.774660 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:06.774682 (+ 22us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:06.775281 (+ 599us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:06.775680 (+ 399us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:06.776815 (+ 1135us) server_negotiation.cc:264] Negotiation successful 0627 19:31:06.777275 (+ 460us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":7671,"thread_start_us":8675,"threads_started":1} I0627 19:31:06.783758 18291 heartbeater.cc:346] Connected to a master server at 127.12.131.62:45576 I0627 19:31:06.784075 18291 heartbeater.cc:426] Registering TS with master... I0627 19:31:06.784559 18291 heartbeater.cc:475] Master 127.12.131.62:45576 requested a full tablet report, sending... I0627 19:31:06.786947 17925 ts_manager.cc:98] Registered new tserver with Master: af2316a3e9f04cab81bf9d17c109f63f (127.12.131.1:33066) I0627 19:31:06.789674 17925 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.1@KRBTEST.COM'} at 127.12.131.1:53730 Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. I0627 19:31:06.804708 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.12.131.2:0 --local_ip_for_outbound_sockets=127.12.131.2 --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:45576 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:07.015430 18313 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:07.015938 18313 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.2 W0627 19:31:07.016372 18313 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:07.033596 18313 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:07.033852 18313 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:07.034162 18313 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.2 W0627 19:31:07.034745 18313 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.2 I0627 19:31:07.075773 18313 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.2:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:45576 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.2 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:07.092023 18313 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:07.094974 18313 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 202592us Jun 27 19:31:07 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153067, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:07.098978 18313 init.cc:339] Logged in from keytab as kudu/127.12.131.2@KRBTEST.COM (short username kudu) I0627 19:31:07.119729 18313 fs_manager.cc:261] Metadata directory not provided I0627 19:31:07.119982 18313 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:07.120404 18313 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:07.120648 18313 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:07.135546 18313 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/data/instance: uuid: "9cbdeae025a241debd86a82881d7ffad" format_stamp: "Formatted at 2018-06-28 02:31:07 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:07.136651 18313 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/wal/instance: uuid: "9cbdeae025a241debd86a82881d7ffad" format_stamp: "Formatted at 2018-06-28 02:31:07 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:07.156960 18313 fs_manager.cc:501] Time spent creating directory manager: real 0.020s user 0.012s sys 0.006s I0627 19:31:07.158954 18313 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:07.159762 18313 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:07.177235 18313 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.001s sys 0.004s I0627 19:31:07.177516 18313 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/wal uuid: "9cbdeae025a241debd86a82881d7ffad" format_stamp: "Formatted at 2018-06-28 02:31:07 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:07.177916 18313 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:07.443235 18313 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:07.446985 18313 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:07.449136 18313 tablet_server_main.cc:90] Starting tablet server... I0627 19:31:07.793467 18291 heartbeater.cc:467] Master 127.12.131.62:45576 was elected leader, sending a full tablet report... I0627 19:31:07.974901 18313 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.2:38308 I0627 19:31:07.976979 18313 webserver.cc:175] Starting webserver on 127.12.131.2:0 I0627 19:31:07.977135 18313 webserver.cc:186] Document root disabled I0627 19:31:07.982681 18313 webserver.cc:313] Webserver started. Bound to: http://127.12.131.2:48053/ I0627 19:31:07.983901 18313 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-1/data/info.pb I0627 19:31:07.993544 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 18313 I0627 19:31:08.008404 18313 tablet_server_main.cc:93] Tablet server successfully started. WARNING: no policy specified for kudu/127.12.131.3@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.3@KRBTEST.COM" created. Jun 27 19:31:08 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153067, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. I0627 19:31:08.057965 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.12.131.3:0 --local_ip_for_outbound_sockets=127.12.131.3 --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:45576 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:31:08.058099 18565 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:07.987799 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:52652 0627 19:31:07.995266 (+ 7467us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:07.995294 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:07.998507 (+ 3213us) server_negotiation.cc:373] Connection header received 0627 19:31:08.021402 (+ 22895us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:08.021437 (+ 35us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:08.021571 (+ 134us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:08.021721 (+ 150us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:08.028667 (+ 6946us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:08.029154 (+ 487us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:08.031129 (+ 1975us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:08.032231 (+ 1102us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:08.032444 (+ 213us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:08.038306 (+ 5862us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:08.038409 (+ 103us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:08.038436 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:08.038520 (+ 84us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:08.046999 (+ 8479us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:08.050680 (+ 3681us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:08.050708 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:08.050727 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:08.051411 (+ 684us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:08.052563 (+ 1152us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:08.052586 (+ 23us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:08.052606 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:08.053057 (+ 451us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:08.053427 (+ 370us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:08.055062 (+ 1635us) server_negotiation.cc:264] Negotiation successful 0627 19:31:08.055642 (+ 580us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5712,"thread_start_us":5819,"threads_started":1} I0627 19:31:08.064460 18566 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:07.987990 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:45576 0627 19:31:07.998256 (+ 10266us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:07.998339 (+ 83us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:08.020983 (+ 22644us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:08.022752 (+ 1769us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:08.022786 (+ 34us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:08.023662 (+ 876us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:08.023957 (+ 295us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:08.024367 (+ 410us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:08.024452 (+ 85us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:08.029669 (+ 5217us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:08.029696 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:08.030909 (+ 1213us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:08.030955 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:08.032638 (+ 1683us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:08.032666 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:08.032980 (+ 314us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:08.033277 (+ 297us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:08.033298 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:08.037746 (+ 4448us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:08.048075 (+ 10329us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:08.048107 (+ 32us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:08.048135 (+ 28us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:08.050193 (+ 2058us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:08.051808 (+ 1615us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:08.051833 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:08.051852 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:08.052113 (+ 261us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:08.053526 (+ 1413us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:08.053554 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:08.054013 (+ 459us) client_negotiation.cc:695] Sending connection context 0627 19:31:08.054565 (+ 552us) client_negotiation.cc:225] Negotiation successful 0627 19:31:08.054972 (+ 407us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":9851,"thread_start_us":13279,"threads_started":1} I0627 19:31:08.067247 18564 heartbeater.cc:346] Connected to a master server at 127.12.131.62:45576 I0627 19:31:08.067607 18564 heartbeater.cc:426] Registering TS with master... I0627 19:31:08.068271 18564 heartbeater.cc:475] Master 127.12.131.62:45576 requested a full tablet report, sending... I0627 19:31:08.070456 17925 ts_manager.cc:98] Registered new tserver with Master: 9cbdeae025a241debd86a82881d7ffad (127.12.131.2:38308) I0627 19:31:08.072289 17925 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.2@KRBTEST.COM'} at 127.12.131.2:52652 WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:08.281824 18570 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:08.282366 18570 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.3 W0627 19:31:08.282879 18570 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:08.300187 18570 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:08.300484 18570 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:08.300791 18570 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.3 W0627 19:31:08.301378 18570 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.3 I0627 19:31:08.346698 18570 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.3:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:45576 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.3 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:08.366869 18570 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:08.370553 18570 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 203092us Jun 27 19:31:08 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153068, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:08.378618 18570 init.cc:339] Logged in from keytab as kudu/127.12.131.3@KRBTEST.COM (short username kudu) I0627 19:31:08.405455 18570 fs_manager.cc:261] Metadata directory not provided I0627 19:31:08.405710 18570 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:08.406121 18570 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:08.406330 18570 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:08.430438 18570 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/data/instance: uuid: "63b2c06401da409da565c4dd12ab6e51" format_stamp: "Formatted at 2018-06-28 02:31:08 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:08.434033 18570 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/wal/instance: uuid: "63b2c06401da409da565c4dd12ab6e51" format_stamp: "Formatted at 2018-06-28 02:31:08 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:08.458138 18570 fs_manager.cc:501] Time spent creating directory manager: real 0.022s user 0.011s sys 0.004s I0627 19:31:08.460182 18570 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:08.461740 18570 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:08.494485 18570 fs_manager.cc:417] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s I0627 19:31:08.494681 18570 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/wal uuid: "63b2c06401da409da565c4dd12ab6e51" format_stamp: "Formatted at 2018-06-28 02:31:08 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:08.495014 18570 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:08.822738 18570 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:08.833580 18570 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:08.835973 18570 tablet_server_main.cc:90] Starting tablet server... I0627 19:31:09.075881 18564 heartbeater.cc:467] Master 127.12.131.62:45576 was elected leader, sending a full tablet report... I0627 19:31:09.439883 18570 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.3:47121 I0627 19:31:09.448133 18570 webserver.cc:175] Starting webserver on 127.12.131.3:0 I0627 19:31:09.448344 18570 webserver.cc:186] Document root disabled I0627 19:31:09.457126 18570 webserver.cc:313] Webserver started. Bound to: http://127.12.131.3:59318/ I0627 19:31:09.458123 18570 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1530153042096250-12812/minicluster-data/ts-2/data/info.pb I0627 19:31:09.459075 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 18570 Jun 27 19:31:09 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:09.494020 18570 tablet_server_main.cc:93] Tablet server successfully started. I0627 19:31:09.497316 18736 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:09.460944 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:49319 0627 19:31:09.465904 (+ 4960us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:09.465928 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:09.470695 (+ 4767us) server_negotiation.cc:373] Connection header received 0627 19:31:09.470816 (+ 121us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:09.470839 (+ 23us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:09.470954 (+ 115us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:09.471098 (+ 144us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:09.473179 (+ 2081us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:09.473663 (+ 484us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:09.475115 (+ 1452us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:09.476170 (+ 1055us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:09.476384 (+ 214us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:09.480948 (+ 4564us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:09.480974 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:09.480996 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:09.481076 (+ 80us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:09.488734 (+ 7658us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:09.490588 (+ 1854us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:09.490614 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:09.490633 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:09.491236 (+ 603us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:09.491812 (+ 576us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:09.491842 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:09.491861 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:09.492298 (+ 437us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:09.494803 (+ 2505us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:09.495069 (+ 266us) server_negotiation.cc:264] Negotiation successful 0627 19:31:09.495534 (+ 465us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":4756,"thread_start_us":4532,"threads_started":1} Jun 27 19:31:09 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153068, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:09.531131 18739 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:09.476751 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:45576 0627 19:31:09.485262 (+ 8511us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:09.485388 (+ 126us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:09.504888 (+ 19500us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:09.505655 (+ 767us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:09.505687 (+ 32us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:09.506500 (+ 813us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:09.506789 (+ 289us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:09.507088 (+ 299us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:09.507136 (+ 48us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:09.507827 (+ 691us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:09.507852 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:09.509072 (+ 1220us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:09.509120 (+ 48us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:09.510174 (+ 1054us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:09.510202 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:09.510544 (+ 342us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:09.510895 (+ 351us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:09.510915 (+ 20us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:09.513983 (+ 3068us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:09.522092 (+ 8109us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:09.522148 (+ 56us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:09.522177 (+ 29us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:09.523413 (+ 1236us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:09.525266 (+ 1853us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:09.525292 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:09.525313 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:09.525584 (+ 271us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:09.527361 (+ 1777us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:09.527416 (+ 55us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:09.527833 (+ 417us) client_negotiation.cc:695] Sending connection context 0627 19:31:09.528306 (+ 473us) client_negotiation.cc:225] Negotiation successful 0627 19:31:09.528748 (+ 442us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":8042,"thread_start_us":7731,"threads_started":1} I0627 19:31:09.538054 18740 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:09.477931 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:48317 0627 19:31:09.492620 (+ 14689us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:09.492643 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:09.492681 (+ 38us) server_negotiation.cc:373] Connection header received 0627 19:31:09.505177 (+ 12496us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:09.505205 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:09.505294 (+ 89us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:09.505412 (+ 118us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:09.507311 (+ 1899us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:09.507664 (+ 353us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:09.509287 (+ 1623us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:09.509983 (+ 696us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:09.510143 (+ 160us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:09.514526 (+ 4383us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:09.514602 (+ 76us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:09.514629 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:09.514712 (+ 83us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:09.521603 (+ 6891us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:09.524005 (+ 2402us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:09.524035 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:09.524054 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:09.524729 (+ 675us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:09.526259 (+ 1530us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:09.526284 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:09.526302 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:09.526799 (+ 497us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:09.531827 (+ 5028us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:09.532234 (+ 407us) server_negotiation.cc:264] Negotiation successful 0627 19:31:09.532621 (+ 387us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":9214,"thread_start_us":8961,"threads_started":1} I0627 19:31:09.540333 18738 heartbeater.cc:346] Connected to a master server at 127.12.131.62:45576 I0627 19:31:09.540681 18738 heartbeater.cc:426] Registering TS with master... I0627 19:31:09.541292 18738 heartbeater.cc:475] Master 127.12.131.62:45576 requested a full tablet report, sending... I0627 19:31:09.543215 17925 ts_manager.cc:98] Registered new tserver with Master: 63b2c06401da409da565c4dd12ab6e51 (127.12.131.3:47121) I0627 19:31:09.544999 17925 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.3@KRBTEST.COM'} at 127.12.131.3:48317 I0627 19:31:09.547338 12812 external_mini_cluster.cc:490] 3 TS(s) registered with all masters Jun 27 19:31:09 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: Jun 27 19:31:09 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.1@KRBTEST.COM I0627 19:31:09.635632 18759 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:09.598357 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:51321 0627 19:31:09.608537 (+ 10180us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:09.608560 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:09.608597 (+ 37us) server_negotiation.cc:373] Connection header received 0627 19:31:09.608683 (+ 86us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:09.608699 (+ 16us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:09.608818 (+ 119us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:09.608982 (+ 164us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:09.611183 (+ 2201us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:09.611571 (+ 388us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:09.613080 (+ 1509us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:09.614122 (+ 1042us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:09.614283 (+ 161us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:09.619052 (+ 4769us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:09.619093 (+ 41us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:09.619123 (+ 30us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:09.619205 (+ 82us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:09.626746 (+ 7541us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:09.628730 (+ 1984us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:09.628763 (+ 33us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:09.628784 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:09.629512 (+ 728us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:09.630109 (+ 597us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:09.630135 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:09.630155 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:09.630742 (+ 587us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:09.631124 (+ 382us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:09.633041 (+ 1917us) server_negotiation.cc:264] Negotiation successful 0627 19:31:09.633551 (+ 510us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":9940,"thread_start_us":9647,"threads_started":1} Jun 27 19:31:09 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[17725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:09.677745 18740 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:09.648801 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:49321 0627 19:31:09.649118 (+ 317us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:09.649143 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:09.651162 (+ 2019us) server_negotiation.cc:373] Connection header received 0627 19:31:09.651580 (+ 418us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:09.651607 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:09.651736 (+ 129us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:09.651875 (+ 139us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:09.654132 (+ 2257us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:09.654639 (+ 507us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:09.656166 (+ 1527us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:09.657261 (+ 1095us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:09.657467 (+ 206us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:09.662094 (+ 4627us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:09.662125 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:09.662149 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:09.662237 (+ 88us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:09.668867 (+ 6630us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:09.671428 (+ 2561us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:09.671480 (+ 52us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:09.671502 (+ 22us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:09.672169 (+ 667us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:09.672798 (+ 629us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:09.672824 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:09.672844 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:09.673329 (+ 485us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:09.673653 (+ 324us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:09.675495 (+ 1842us) server_negotiation.cc:264] Negotiation successful 0627 19:31:09.675954 (+ 459us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":91} W0627 19:31:09.678812 17925 server_base.cc:564] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.12.131.62:49321 I0627 19:31:09.680243 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 17987 I0627 19:31:09.730381 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 18313 W0627 19:31:09.733693 17850 connection.cc:512] server connection from 127.12.131.1:53730 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:53730): Connection reset by peer (error 104) W0627 19:31:09.780231 17852 connection.cc:512] server connection from 127.12.131.2:52652 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:52652): Connection reset by peer (error 104) I0627 19:31:09.781229 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 18570 W0627 19:31:09.816656 17852 connection.cc:512] server connection from 127.12.131.3:48317 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.3:48317): Connection reset by peer (error 104) I0627 19:31:09.817546 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 17774 [ OK ] SecurityITest.TestAuthorizedSuperuser (6940 ms) [ RUN ] SecurityITest.TestDisableWebUI Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:10 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): setting up network... Jun 27 19:31:10 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:10 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): skipping unrecognized local address family 17 Jun 27 19:31:10 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): skipping unrecognized local address family 17 Jun 27 19:31:10 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): set up 1 sockets Jun 27 19:31:10 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): commencing operation krb5kdc: starting... W0627 19:31:10.137872 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.266s user 0.003s sys 0.014s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:10 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:31:10.459951 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:51018 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --webserver_enabled=0 --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:10.765336 18830 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:10.765866 18830 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:31:10.766304 18830 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:10.784195 18830 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:10.784494 18830 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:10.784690 18830 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:10.784883 18830 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:10.785488 18830 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:10.785732 18830 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:31:10.823958 18830 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:51018 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --webserver_enabled=false --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:10.848934 18830 master_main.cc:80] Initializing master server... I0627 19:31:10.853619 18830 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 204092us Jun 27 19:31:10 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153070, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:10.865486 18830 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:31:10.879554 18830 fs_manager.cc:261] Metadata directory not provided I0627 19:31:10.879804 18830 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:10.880223 18830 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:10.880425 18830 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:10.896705 18830 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" format_stamp: "Formatted at 2018-06-28 02:31:10 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:10.897827 18830 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" format_stamp: "Formatted at 2018-06-28 02:31:10 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:10.921159 18830 fs_manager.cc:501] Time spent creating directory manager: real 0.023s user 0.013s sys 0.006s I0627 19:31:10.923367 18830 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:10.924962 18830 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:10.943307 18830 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.001s sys 0.004s I0627 19:31:10.943580 18830 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/wal uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" format_stamp: "Formatted at 2018-06-28 02:31:10 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:10.943958 18830 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:11.212569 18830 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:11.220800 18830 master_main.cc:83] Starting Master server... I0627 19:31:11.652336 18830 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:51018 I0627 19:31:11.653380 18830 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:11.655345 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 18830 I0627 19:31:11.662308 19009 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed WARNING: no policy specified for kudu/127.12.131.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.1@KRBTEST.COM" created. I0627 19:31:11.689327 19009 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17: Bootstrap starting. I0627 19:31:11.692538 19009 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17: No blocks or log segments found. Creating new log. I0627 19:31:11.695003 19009 log.cc:525] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17: Log is configured to *not* fsync() on all Append() calls I0627 19:31:11.703544 19009 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17: No bootstrap required, opened a new log Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. I0627 19:31:11.726114 19009 raft_consensus.cc:325] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" member_type: VOTER } I0627 19:31:11.726760 19009 raft_consensus.cc:351] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:11.727022 19009 raft_consensus.cc:628] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb35f880a5c14c5a8a94dcc34bef3a17, State: Initialized, Role: FOLLOWER I0627 19:31:11.727252 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.12.131.1:0 --local_ip_for_outbound_sockets=127.12.131.1 --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:51018 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --webserver_enabled=0 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:31:11.728230 19009 consensus_queue.cc:227] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" member_type: VOTER } I0627 19:31:11.728818 19009 raft_consensus.cc:365] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:11.729193 19009 raft_consensus.cc:457] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:11.729496 19009 raft_consensus.cc:2730] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17: Snoozing failure detection for 1.950s (starting election) I0627 19:31:11.729773 19009 raft_consensus.cc:2782] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:11.734994 19009 raft_consensus.cc:479] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" member_type: VOTER } I0627 19:31:11.735821 19009 leader_election.cc:270] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:11.745558 19017 raft_consensus.cc:2730] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17: Snoozing failure detection for 1.512s (election complete) I0627 19:31:11.745880 19017 raft_consensus.cc:2537] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:11.754503 19017 raft_consensus.cc:593] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [term 1 LEADER]: Becoming Leader. State: Replica: cb35f880a5c14c5a8a94dcc34bef3a17, State: Running, Role: LEADER I0627 19:31:11.755014 19017 consensus_queue.cc:204] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" member_type: VOTER } I0627 19:31:11.768505 19009 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" member_type: VOTER } } I0627 19:31:11.769019 19009 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [sys.catalog]: This master's current role is: LEADER I0627 19:31:11.768857 19018 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb35f880a5c14c5a8a94dcc34bef3a17. Latest consensus state: current_term: 1 leader_uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" member_type: VOTER } } I0627 19:31:12.002975 19018 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [sys.catalog]: This master's current role is: LEADER I0627 19:31:11.769011 19019 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb35f880a5c14c5a8a94dcc34bef3a17" member_type: VOTER } } I0627 19:31:12.008287 19019 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [sys.catalog]: This master's current role is: LEADER I0627 19:31:12.015703 19009 sys_catalog.cc:424] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:12.025490 18830 master_main.cc:86] Master server successfully started. W0627 19:31:12.046702 19027 catalog_manager.cc:1064] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:12.046955 19027 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:31:12.047296 19025 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:12.055316 19025 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:12.069559 19016 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:12.077729 19016 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.1 W0627 19:31:12.078228 19016 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:12.107672 19016 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:12.107945 19016 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:12.108314 19016 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.1 W0627 19:31:12.108929 19016 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.1 I0627 19:31:12.134634 19025 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:12.136996 19025 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:12.164253 19025 catalog_manager.cc:4102] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17: Generated new TSK 0 I0627 19:31:12.177382 19016 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.1:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --webserver_enabled=false --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:51018 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:12.204563 19016 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:12.210803 19016 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 205092us Jun 27 19:31:12 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153072, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:12.215212 19016 init.cc:339] Logged in from keytab as kudu/127.12.131.1@KRBTEST.COM (short username kudu) I0627 19:31:12.249924 19016 fs_manager.cc:261] Metadata directory not provided I0627 19:31:12.250177 19016 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:12.250617 19016 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:12.251569 19016 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:12.280133 19016 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/data/instance: uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" format_stamp: "Formatted at 2018-06-28 02:31:12 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:12.281625 19016 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/wal/instance: uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" format_stamp: "Formatted at 2018-06-28 02:31:12 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:12.319190 19016 fs_manager.cc:501] Time spent creating directory manager: real 0.037s user 0.014s sys 0.003s I0627 19:31:12.321118 19016 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:12.324679 19016 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:12.350200 19016 fs_manager.cc:417] Time spent opening block manager: real 0.010s user 0.001s sys 0.002s I0627 19:31:12.350466 19016 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/wal uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" format_stamp: "Formatted at 2018-06-28 02:31:12 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:12.350836 19016 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:12.669564 19016 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:12.680497 19016 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:12.681877 19016 tablet_server_main.cc:90] Starting tablet server... I0627 19:31:13.211588 19016 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.1:46489 I0627 19:31:13.212558 19016 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-0/data/info.pb I0627 19:31:13.217712 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 19016 I0627 19:31:13.236829 19016 tablet_server_main.cc:93] Tablet server successfully started. WARNING: no policy specified for kudu/127.12.131.2@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.2@KRBTEST.COM" created. Jun 27 19:31:13 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153072, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. I0627 19:31:13.264605 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.12.131.2:0 --local_ip_for_outbound_sockets=127.12.131.2 --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:51018 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --webserver_enabled=0 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:31:13.286365 19276 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:13.218098 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:51018 0627 19:31:13.235216 (+ 17118us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:13.235362 (+ 146us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:13.251292 (+ 15930us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:13.252836 (+ 1544us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:13.252868 (+ 32us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:13.253719 (+ 851us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:13.254015 (+ 296us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:13.254472 (+ 457us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:13.254523 (+ 51us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:13.255674 (+ 1151us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:13.255699 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:13.256732 (+ 1033us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:13.256780 (+ 48us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:13.258434 (+ 1654us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:13.258463 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:13.258762 (+ 299us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:13.258959 (+ 197us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:13.258971 (+ 12us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:13.261489 (+ 2518us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:13.277276 (+ 15787us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:13.277298 (+ 22us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:13.277312 (+ 14us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:13.278232 (+ 920us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:13.279748 (+ 1516us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:13.279773 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:13.279793 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:13.280021 (+ 228us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:13.281923 (+ 1902us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:13.281951 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:13.282365 (+ 414us) client_negotiation.cc:695] Sending connection context 0627 19:31:13.282860 (+ 495us) client_negotiation.cc:225] Negotiation successful 0627 19:31:13.283243 (+ 383us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":16206,"thread_start_us":7028,"threads_started":1} I0627 19:31:13.290356 19277 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:13.220708 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:55021 0627 19:31:13.226808 (+ 6100us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:13.226847 (+ 39us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:13.235596 (+ 8749us) server_negotiation.cc:373] Connection header received 0627 19:31:13.252027 (+ 16431us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:13.252073 (+ 46us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:13.252320 (+ 247us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:13.252669 (+ 349us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:13.254721 (+ 2052us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:13.255460 (+ 739us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:13.256998 (+ 1538us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:13.258202 (+ 1204us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:13.258395 (+ 193us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:13.261993 (+ 3598us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:13.262047 (+ 54us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:13.262066 (+ 19us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:13.262122 (+ 56us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:13.276708 (+ 14586us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:13.278641 (+ 1933us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:13.278671 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:13.278690 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:13.279329 (+ 639us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:13.280744 (+ 1415us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:13.280769 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:13.280787 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:13.281348 (+ 561us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:13.287015 (+ 5667us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:13.287584 (+ 569us) server_negotiation.cc:264] Negotiation successful 0627 19:31:13.288008 (+ 424us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5735,"thread_start_us":5242,"threads_started":1} I0627 19:31:13.293611 19274 heartbeater.cc:346] Connected to a master server at 127.12.131.62:51018 I0627 19:31:13.293892 19274 heartbeater.cc:426] Registering TS with master... I0627 19:31:13.294356 19274 heartbeater.cc:475] Master 127.12.131.62:51018 requested a full tablet report, sending... I0627 19:31:13.296447 18973 ts_manager.cc:98] Registered new tserver with Master: 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) I0627 19:31:13.299345 18973 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.1@KRBTEST.COM'} at 127.12.131.1:55021 WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:13.402595 19280 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:13.402928 19280 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.2 W0627 19:31:13.403219 19280 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:13.412489 19280 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:13.412638 19280 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:13.412832 19280 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.2 W0627 19:31:13.413182 19280 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.2 I0627 19:31:13.432070 19280 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.2:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --webserver_enabled=false --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:51018 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.2 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:13.441042 19280 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:13.442705 19280 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 205592us Jun 27 19:31:13 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153073, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:13.444902 19280 init.cc:339] Logged in from keytab as kudu/127.12.131.2@KRBTEST.COM (short username kudu) I0627 19:31:13.448056 19280 fs_manager.cc:261] Metadata directory not provided I0627 19:31:13.448197 19280 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:13.448493 19280 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:13.448645 19280 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:13.457988 19280 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/data/instance: uuid: "07e4d1e7f5c3453fb7f387b131ea8853" format_stamp: "Formatted at 2018-06-28 02:31:13 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:13.458740 19280 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/wal/instance: uuid: "07e4d1e7f5c3453fb7f387b131ea8853" format_stamp: "Formatted at 2018-06-28 02:31:13 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:13.478456 19280 fs_manager.cc:501] Time spent creating directory manager: real 0.019s user 0.011s sys 0.007s I0627 19:31:13.480195 19280 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:13.480792 19280 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:13.497829 19280 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.000s sys 0.006s I0627 19:31:13.498060 19280 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/wal uuid: "07e4d1e7f5c3453fb7f387b131ea8853" format_stamp: "Formatted at 2018-06-28 02:31:13 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:13.498427 19280 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:13.738903 19280 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:13.749339 19280 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:13.750656 19280 tablet_server_main.cc:90] Starting tablet server... I0627 19:31:14.302439 19274 heartbeater.cc:467] Master 127.12.131.62:51018 was elected leader, sending a full tablet report... I0627 19:31:14.537937 19280 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.2:60974 I0627 19:31:14.539306 19280 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/data/info.pb I0627 19:31:14.548194 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 19280 I0627 19:31:14.567945 19280 tablet_server_main.cc:93] Tablet server successfully started. WARNING: no policy specified for kudu/127.12.131.3@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.3@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Jun 27 19:31:14 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153073, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:14.615372 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.12.131.3:0 --local_ip_for_outbound_sockets=127.12.131.3 --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:51018 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --webserver_enabled=0 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:31:14.653795 19481 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:14.556494 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:51018 0627 19:31:14.570225 (+ 13731us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:14.570302 (+ 77us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:14.595344 (+ 25042us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:14.604772 (+ 9428us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:14.604814 (+ 42us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:14.605658 (+ 844us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:14.605959 (+ 301us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:14.606380 (+ 421us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:14.606453 (+ 73us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:14.607493 (+ 1040us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:14.607520 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:14.608750 (+ 1230us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:14.608796 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:14.611289 (+ 2493us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:14.611321 (+ 32us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:14.611680 (+ 359us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:14.611979 (+ 299us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:14.612001 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:14.617634 (+ 5633us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:14.634473 (+ 16839us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:14.634508 (+ 35us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:14.634537 (+ 29us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:14.638723 (+ 4186us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:14.643515 (+ 4792us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:14.643547 (+ 32us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:14.643568 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:14.643826 (+ 258us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:14.648348 (+ 4522us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:14.648371 (+ 23us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:14.648734 (+ 363us) client_negotiation.cc:695] Sending connection context 0627 19:31:14.649113 (+ 379us) client_negotiation.cc:225] Negotiation successful 0627 19:31:14.649567 (+ 454us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":13373,"thread_start_us":14513,"threads_started":1} I0627 19:31:14.656718 19482 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:14.557810 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:52315 0627 19:31:14.577429 (+ 19619us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:14.577476 (+ 47us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:14.577516 (+ 40us) server_negotiation.cc:373] Connection header received 0627 19:31:14.595691 (+ 18175us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:14.595723 (+ 32us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:14.595850 (+ 127us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:14.596022 (+ 172us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:14.606669 (+ 10647us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:14.607266 (+ 597us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:14.609056 (+ 1790us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:14.610174 (+ 1118us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:14.610346 (+ 172us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:14.618405 (+ 8059us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:14.618485 (+ 80us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:14.618512 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:14.618620 (+ 108us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:14.631714 (+ 13094us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:14.639319 (+ 7605us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:14.639348 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:14.639367 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:14.641536 (+ 2169us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:14.645852 (+ 4316us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:14.645883 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:14.645903 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:14.646467 (+ 564us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:14.646818 (+ 351us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:14.650929 (+ 4111us) server_negotiation.cc:264] Negotiation successful 0627 19:31:14.651342 (+ 413us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":19417,"thread_start_us":17184,"threads_started":1} I0627 19:31:14.662923 19478 heartbeater.cc:346] Connected to a master server at 127.12.131.62:51018 I0627 19:31:14.663249 19478 heartbeater.cc:426] Registering TS with master... I0627 19:31:14.663838 19478 heartbeater.cc:475] Master 127.12.131.62:51018 requested a full tablet report, sending... I0627 19:31:14.666532 18973 ts_manager.cc:98] Registered new tserver with Master: 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974) I0627 19:31:14.668349 18973 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.2@KRBTEST.COM'} at 127.12.131.2:52315 WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:14.840590 19484 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:14.841099 19484 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.3 W0627 19:31:14.841569 19484 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:14.867498 19484 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:14.875552 19484 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:14.875913 19484 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.3 W0627 19:31:14.876520 19484 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.3 I0627 19:31:14.940502 19484 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.3:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --webserver_enabled=false --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:51018 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.3 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:14.981163 19484 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:14.984746 19484 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 206092us Jun 27 19:31:14 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153074, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:14.999774 19484 init.cc:339] Logged in from keytab as kudu/127.12.131.3@KRBTEST.COM (short username kudu) I0627 19:31:15.011112 19484 fs_manager.cc:261] Metadata directory not provided I0627 19:31:15.011303 19484 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:15.011679 19484 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:15.011848 19484 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:15.027930 19484 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/data/instance: uuid: "551f9818aeda4b50af2d9ac4e99b5380" format_stamp: "Formatted at 2018-06-28 02:31:15 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:15.029124 19484 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/wal/instance: uuid: "551f9818aeda4b50af2d9ac4e99b5380" format_stamp: "Formatted at 2018-06-28 02:31:15 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:15.049388 19484 fs_manager.cc:501] Time spent creating directory manager: real 0.020s user 0.013s sys 0.004s I0627 19:31:15.051383 19484 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:15.052193 19484 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:15.070252 19484 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.001s sys 0.004s I0627 19:31:15.070519 19484 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/wal uuid: "551f9818aeda4b50af2d9ac4e99b5380" format_stamp: "Formatted at 2018-06-28 02:31:15 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:15.070900 19484 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:15.279129 19484 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:15.284673 19484 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:15.287848 19484 tablet_server_main.cc:90] Starting tablet server... I0627 19:31:15.672010 19478 heartbeater.cc:467] Master 127.12.131.62:51018 was elected leader, sending a full tablet report... I0627 19:31:16.064167 19484 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.3:57511 I0627 19:31:16.065263 19484 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/data/info.pb I0627 19:31:16.072703 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 19484 I0627 19:31:16.093927 19484 tablet_server_main.cc:93] Tablet server successfully started. Jun 27 19:31:16 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:16.123282 19642 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.074037 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:53846 0627 19:31:16.085003 (+ 10966us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:16.085030 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:16.085070 (+ 40us) server_negotiation.cc:373] Connection header received 0627 19:31:16.085187 (+ 117us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:16.085212 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:16.085336 (+ 124us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:16.085516 (+ 180us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:16.087864 (+ 2348us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.088327 (+ 463us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.097632 (+ 9305us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.098725 (+ 1093us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.098912 (+ 187us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:16.103688 (+ 4776us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:16.103719 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:16.103741 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:16.103822 (+ 81us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:16.114204 (+ 10382us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:16.116256 (+ 2052us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:16.116287 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:16.116306 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:16.116972 (+ 666us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:16.118233 (+ 1261us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:16.118259 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:16.118282 (+ 23us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:16.118765 (+ 483us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:16.119133 (+ 368us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:16.120998 (+ 1865us) server_negotiation.cc:264] Negotiation successful 0627 19:31:16.121466 (+ 468us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":10749,"thread_start_us":11327,"threads_started":1} Jun 27 19:31:16 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[18808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153074, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:16.179002 19645 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.086321 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:42028 0627 19:31:16.118065 (+ 31744us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:16.118090 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:16.118127 (+ 37us) server_negotiation.cc:373] Connection header received 0627 19:31:16.147975 (+ 29848us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:16.148003 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:16.148121 (+ 118us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:16.148259 (+ 138us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:16.150904 (+ 2645us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.151357 (+ 453us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.153137 (+ 1780us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.154195 (+ 1058us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.154366 (+ 171us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:16.160850 (+ 6484us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:16.160923 (+ 73us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:16.160949 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:16.161037 (+ 88us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:16.167969 (+ 6932us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:16.171070 (+ 3101us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:16.171100 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:16.171121 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:16.171811 (+ 690us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:16.173281 (+ 1470us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:16.173309 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:16.173329 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:16.173840 (+ 511us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:16.174181 (+ 341us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:16.176825 (+ 2644us) server_negotiation.cc:264] Negotiation successful 0627 19:31:16.177182 (+ 357us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":31529,"thread_start_us":33150,"threads_started":1} I0627 19:31:16.182301 19644 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.085014 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:51018 0627 19:31:16.089059 (+ 4045us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:16.089179 (+ 120us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:16.147656 (+ 58477us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:16.148875 (+ 1219us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:16.148911 (+ 36us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:16.149834 (+ 923us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:16.150156 (+ 322us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:16.150614 (+ 458us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:16.150673 (+ 59us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.151681 (+ 1008us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.151708 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:16.152886 (+ 1178us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:16.152930 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.154847 (+ 1917us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.154878 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:16.155198 (+ 320us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:16.155534 (+ 336us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:16.155555 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:16.160090 (+ 4535us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:16.168787 (+ 8697us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:16.168819 (+ 32us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:16.168848 (+ 29us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:16.170464 (+ 1616us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:16.172367 (+ 1903us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:16.172395 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:16.172446 (+ 51us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:16.172720 (+ 274us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:16.174559 (+ 1839us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:16.174590 (+ 31us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:16.175478 (+ 888us) client_negotiation.cc:695] Sending connection context 0627 19:31:16.176034 (+ 556us) client_negotiation.cc:225] Negotiation successful 0627 19:31:16.179854 (+ 3820us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":3477,"thread_start_us":3099,"threads_started":1} I0627 19:31:16.191736 19640 heartbeater.cc:346] Connected to a master server at 127.12.131.62:51018 I0627 19:31:16.192137 19640 heartbeater.cc:426] Registering TS with master... I0627 19:31:16.192870 19640 heartbeater.cc:475] Master 127.12.131.62:51018 requested a full tablet report, sending... I0627 19:31:16.194878 18973 ts_manager.cc:98] Registered new tserver with Master: 551f9818aeda4b50af2d9ac4e99b5380 (127.12.131.3:57511) I0627 19:31:16.196512 12812 external_mini_cluster.cc:490] 3 TS(s) registered with all masters I0627 19:31:16.196715 18973 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.3@KRBTEST.COM'} at 127.12.131.3:42028 I0627 19:31:16.264922 19645 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.228596 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:53847 0627 19:31:16.228938 (+ 342us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:16.228961 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:16.232051 (+ 3090us) server_negotiation.cc:373] Connection header received 0627 19:31:16.232804 (+ 753us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:16.232832 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:16.232969 (+ 137us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:16.233110 (+ 141us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:16.235629 (+ 2519us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.236185 (+ 556us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.238397 (+ 2212us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.239430 (+ 1033us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.239635 (+ 205us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:16.248190 (+ 8555us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:16.248220 (+ 30us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:16.248243 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:16.248318 (+ 75us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:16.254569 (+ 6251us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:16.256491 (+ 1922us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:16.256521 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:16.256540 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:16.257982 (+ 1442us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:16.258582 (+ 600us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:16.258609 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:16.258628 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:16.259122 (+ 494us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:16.259426 (+ 304us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:16.261766 (+ 2340us) server_negotiation.cc:264] Negotiation successful 0627 19:31:16.262800 (+ 1034us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":135} I0627 19:31:16.271423 18973 catalog_manager.cc:1339] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.12.131.62:53847: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "val" type: INT32 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0627 19:31:16.286860 18973 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0627 19:31:16.416100 19662 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.377432 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.2:60974 0627 19:31:16.388443 (+ 11011us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:16.388629 (+ 186us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:16.388760 (+ 131us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:16.392486 (+ 3726us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:16.392517 (+ 31us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:16.392580 (+ 63us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:31:16.393057 (+ 477us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:16.393247 (+ 190us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:16.393285 (+ 38us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.400012 (+ 6727us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.400044 (+ 32us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:16.407746 (+ 7702us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:16.407838 (+ 92us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.411481 (+ 3643us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.411512 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:16.411895 (+ 383us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:16.412078 (+ 183us) client_negotiation.cc:695] Sending connection context 0627 19:31:16.412213 (+ 135us) client_negotiation.cc:225] Negotiation successful 0627 19:31:16.412532 (+ 319us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":10692,"mutex_wait_us":5538,"thread_start_us":15137,"threads_started":1} I0627 19:31:16.416939 19664 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.378003 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:48175 0627 19:31:16.391313 (+ 13310us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:16.391346 (+ 33us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:16.391396 (+ 50us) server_negotiation.cc:373] Connection header received 0627 19:31:16.391570 (+ 174us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:16.391597 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:16.391776 (+ 179us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:16.392109 (+ 333us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:31:16.399226 (+ 7117us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.399833 (+ 607us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.408109 (+ 8276us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.410638 (+ 2529us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.410858 (+ 220us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:16.411192 (+ 334us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:16.415397 (+ 4205us) server_negotiation.cc:264] Negotiation successful 0627 19:31:16.415718 (+ 321us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":13079,"thread_start_us":13725,"threads_started":1} I0627 19:31:16.419817 19388 tablet_service.cc:759] Processing CreateTablet for tablet 32014c9a31f14216874c40a94fc74717 (table=test-table [id=dc0c540c90bf46f4853978eba9ea3ded]), partition=RANGE (key) PARTITION UNBOUNDED I0627 19:31:16.420846 19388 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 32014c9a31f14216874c40a94fc74717. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:16.438830 19666 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.380418 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:41986 0627 19:31:16.387176 (+ 6758us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:16.387210 (+ 34us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:16.397566 (+ 10356us) server_negotiation.cc:373] Connection header received 0627 19:31:16.397972 (+ 406us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:16.398002 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:16.398208 (+ 206us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:16.398353 (+ 145us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:31:16.406865 (+ 8512us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.407440 (+ 575us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.432220 (+ 24780us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.434763 (+ 2543us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.434980 (+ 217us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:16.435342 (+ 362us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:16.437345 (+ 2003us) server_negotiation.cc:264] Negotiation successful 0627 19:31:16.437640 (+ 295us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":6491,"thread_start_us":6179,"threads_started":1} I0627 19:31:16.438865 19661 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.371572 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:34899 0627 19:31:16.375296 (+ 3724us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:16.375325 (+ 29us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:16.385966 (+ 10641us) server_negotiation.cc:373] Connection header received 0627 19:31:16.397817 (+ 11851us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:16.397855 (+ 38us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:16.398081 (+ 226us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:16.398253 (+ 172us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:31:16.411715 (+ 13462us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.412479 (+ 764us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.432665 (+ 20186us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.435195 (+ 2530us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.435414 (+ 219us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:16.435761 (+ 347us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:16.436787 (+ 1026us) server_negotiation.cc:264] Negotiation successful 0627 19:31:16.437056 (+ 269us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":3393,"thread_start_us":3387,"threads_started":1} I0627 19:31:16.440146 19663 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.380247 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.1:46489 0627 19:31:16.397276 (+ 17029us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:16.397447 (+ 171us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:16.397591 (+ 144us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:16.398714 (+ 1123us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:16.398742 (+ 28us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:16.398884 (+ 142us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:31:16.406149 (+ 7265us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:16.406545 (+ 396us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:16.406588 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.407691 (+ 1103us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.407718 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:16.431774 (+ 24056us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:16.431825 (+ 51us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.435562 (+ 3737us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.435590 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:16.435919 (+ 329us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:16.436046 (+ 127us) client_negotiation.cc:695] Sending connection context 0627 19:31:16.436155 (+ 109us) client_negotiation.cc:225] Negotiation successful 0627 19:31:16.436476 (+ 321us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":10257,"mutex_wait_us":25458,"spinlock_wait_cycles":35712,"thread_start_us":5167,"threads_started":1} I0627 19:31:16.443119 19210 tablet_service.cc:759] Processing CreateTablet for tablet 32014c9a31f14216874c40a94fc74717 (table=test-table [id=dc0c540c90bf46f4853978eba9ea3ded]), partition=RANGE (key) PARTITION UNBOUNDED I0627 19:31:16.444180 19210 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 32014c9a31f14216874c40a94fc74717. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:16.458742 19665 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.371307 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.3:57511 0627 19:31:16.385682 (+ 14375us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:16.385779 (+ 97us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:16.386549 (+ 770us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:16.398706 (+ 12157us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:16.398736 (+ 30us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:16.398797 (+ 61us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:31:16.406206 (+ 7409us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:16.407973 (+ 1767us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:16.411451 (+ 3478us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.413626 (+ 2175us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.413723 (+ 97us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:16.432314 (+ 18591us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:16.432360 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:16.436017 (+ 3657us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:16.436044 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:16.436374 (+ 330us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:16.436518 (+ 144us) client_negotiation.cc:695] Sending connection context 0627 19:31:16.436619 (+ 101us) client_negotiation.cc:225] Negotiation successful 0627 19:31:16.457284 (+ 20665us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":13984,"mutex_wait_us":1484,"thread_start_us":15763,"threads_started":1} I0627 19:31:16.462625 19555 tablet_service.cc:759] Processing CreateTablet for tablet 32014c9a31f14216874c40a94fc74717 (table=test-table [id=dc0c540c90bf46f4853978eba9ea3ded]), partition=RANGE (key) PARTITION UNBOUNDED I0627 19:31:16.463654 19555 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 32014c9a31f14216874c40a94fc74717. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:16.485298 19210 ts_tablet_manager.cc:1173] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Registered tablet (data state: TABLET_DATA_READY) I0627 19:31:16.496244 19388 ts_tablet_manager.cc:1173] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Registered tablet (data state: TABLET_DATA_READY) I0627 19:31:16.512246 19672 ts_tablet_manager.cc:1028] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Bootstrapping tablet I0627 19:31:16.513715 19672 tablet_bootstrap.cc:427] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Bootstrap starting. I0627 19:31:16.519851 19672 tablet_bootstrap.cc:578] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: No blocks or log segments found. Creating new log. I0627 19:31:16.529007 19672 log.cc:525] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Log is configured to *not* fsync() on all Append() calls I0627 19:31:16.544147 19671 ts_tablet_manager.cc:1028] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Bootstrapping tablet I0627 19:31:16.547740 19555 ts_tablet_manager.cc:1173] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Registered tablet (data state: TABLET_DATA_READY) I0627 19:31:16.554193 19672 tablet_bootstrap.cc:427] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: No bootstrap required, opened a new log I0627 19:31:16.554719 19672 ts_tablet_manager.cc:1045] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.007s I0627 19:31:16.559038 19671 tablet_bootstrap.cc:427] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Bootstrap starting. I0627 19:31:16.577096 19675 ts_tablet_manager.cc:1028] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Bootstrapping tablet I0627 19:31:16.577833 19675 tablet_bootstrap.cc:427] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Bootstrap starting. I0627 19:31:16.581432 19671 tablet_bootstrap.cc:578] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: No blocks or log segments found. Creating new log. I0627 19:31:16.582502 19675 tablet_bootstrap.cc:578] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: No blocks or log segments found. Creating new log. I0627 19:31:16.583189 19671 log.cc:525] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Log is configured to *not* fsync() on all Append() calls I0627 19:31:16.592072 19675 log.cc:525] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Log is configured to *not* fsync() on all Append() calls I0627 19:31:16.613235 19675 tablet_bootstrap.cc:427] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: No bootstrap required, opened a new log I0627 19:31:16.613737 19675 ts_tablet_manager.cc:1045] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.007s I0627 19:31:16.618434 19671 tablet_bootstrap.cc:427] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: No bootstrap required, opened a new log I0627 19:31:16.618844 19671 ts_tablet_manager.cc:1045] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Time spent bootstrapping tablet: real 0.074s user 0.007s sys 0.001s I0627 19:31:16.640117 19672 raft_consensus.cc:325] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:16.641139 19672 raft_consensus.cc:351] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:16.641412 19672 raft_consensus.cc:628] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464e3b99b46c42f4a9d7c715d84bdf8e, State: Initialized, Role: FOLLOWER I0627 19:31:16.642263 19672 consensus_queue.cc:227] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:16.670980 19672 ts_tablet_manager.cc:1073] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Time spent starting tablet: real 0.116s user 0.019s sys 0.005s I0627 19:31:16.678823 19671 raft_consensus.cc:325] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:16.681551 19671 raft_consensus.cc:351] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:16.681792 19671 raft_consensus.cc:628] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e4d1e7f5c3453fb7f387b131ea8853, State: Initialized, Role: FOLLOWER I0627 19:31:16.685518 19671 consensus_queue.cc:227] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:16.692493 19675 raft_consensus.cc:325] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:16.699928 19671 ts_tablet_manager.cc:1073] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Time spent starting tablet: real 0.081s user 0.019s sys 0.005s I0627 19:31:16.708413 19675 raft_consensus.cc:351] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:16.708699 19675 raft_consensus.cc:628] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 551f9818aeda4b50af2d9ac4e99b5380, State: Initialized, Role: FOLLOWER I0627 19:31:16.709455 19675 consensus_queue.cc:227] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:16.737138 19640 heartbeater.cc:467] Master 127.12.131.62:51018 was elected leader, sending a full tablet report... I0627 19:31:16.738713 19675 ts_tablet_manager.cc:1073] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Time spent starting tablet: real 0.125s user 0.018s sys 0.006s I0627 19:31:16.854370 19679 raft_consensus.cc:457] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0627 19:31:16.854707 19679 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Snoozing failure detection for 1.671s (starting election) I0627 19:31:16.855027 19679 raft_consensus.cc:479] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:16.856545 19679 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 551f9818aeda4b50af2d9ac4e99b5380 (127.12.131.3:57511) I0627 19:31:16.857113 19679 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) W0627 19:31:16.845634 19641 tablet.cc:1678] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:17.048724 19680 raft_consensus.cc:457] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0627 19:31:17.049036 19680 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Snoozing failure detection for 1.897s (starting election) I0627 19:31:17.049338 19680 raft_consensus.cc:479] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:17.050771 19680 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974) I0627 19:31:17.051440 19680 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) I0627 19:31:17.066421 19678 raft_consensus.cc:457] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0627 19:31:17.066725 19678 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Snoozing failure detection for 1.977s (starting election) I0627 19:31:17.067035 19678 raft_consensus.cc:479] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:17.068552 19678 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 551f9818aeda4b50af2d9ac4e99b5380 (127.12.131.3:57511) I0627 19:31:17.069161 19678 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974) W0627 19:31:16.748997 19275 tablet.cc:1678] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:17.119518 19706 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:17.069886 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.3:57511 0627 19:31:17.102068 (+ 32182us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:17.103079 (+ 1011us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:17.103475 (+ 396us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:17.107661 (+ 4186us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:17.107689 (+ 28us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:17.107746 (+ 57us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:31:17.107981 (+ 235us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.108299 (+ 318us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:17.108340 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.112674 (+ 4334us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.112700 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.114609 (+ 1909us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:17.114654 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.117208 (+ 2554us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.117235 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.117558 (+ 323us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:17.117703 (+ 145us) client_negotiation.cc:695] Sending connection context 0627 19:31:17.117958 (+ 255us) client_negotiation.cc:225] Negotiation successful 0627 19:31:17.118018 (+ 60us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":31763,"thread_start_us":35533,"threads_started":1} I0627 19:31:17.119666 19705 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:17.070800 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:43381 0627 19:31:17.073494 (+ 2694us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:17.073518 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:17.103266 (+ 29748us) server_negotiation.cc:373] Connection header received 0627 19:31:17.103753 (+ 487us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:17.103780 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:17.103922 (+ 142us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:17.104042 (+ 120us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:31:17.108521 (+ 4479us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.109150 (+ 629us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.114867 (+ 5717us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.116789 (+ 1922us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.116997 (+ 208us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:17.117271 (+ 274us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:17.118246 (+ 975us) server_negotiation.cc:264] Negotiation successful 0627 19:31:17.118324 (+ 78us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":2469,"thread_start_us":1337,"threads_started":1} I0627 19:31:17.122375 19700 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:17.052238 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.1:46489 0627 19:31:17.067700 (+ 15462us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:17.067748 (+ 48us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:17.067914 (+ 166us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:17.104789 (+ 36875us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:17.104817 (+ 28us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:17.104875 (+ 58us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:31:17.105118 (+ 243us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.105299 (+ 181us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:17.105336 (+ 37us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.107213 (+ 1877us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.107239 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.109244 (+ 2005us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:17.109290 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.117904 (+ 8614us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.117937 (+ 33us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.120275 (+ 2338us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:17.120461 (+ 186us) client_negotiation.cc:695] Sending connection context 0627 19:31:17.120734 (+ 273us) client_negotiation.cc:225] Negotiation successful 0627 19:31:17.120799 (+ 65us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":15227,"mutex_wait_us":1954,"thread_start_us":2512,"threads_started":1} I0627 19:31:17.129896 19582 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "551f9818aeda4b50af2d9ac4e99b5380" is_pre_election: true I0627 19:31:17.130492 19582 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Snoozing failure detection for 1.837s (vote granted) I0627 19:31:17.130750 19582 raft_consensus.cc:2243] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 464e3b99b46c42f4a9d7c715d84bdf8e in term 0. I0627 19:31:17.122475 19701 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:17.053352 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:38282 0627 19:31:17.104254 (+ 50902us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:17.104282 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:17.104321 (+ 39us) server_negotiation.cc:373] Connection header received 0627 19:31:17.104451 (+ 130us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:17.104478 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:17.104602 (+ 124us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:17.104731 (+ 129us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:31:17.106128 (+ 1397us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.106756 (+ 628us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.110127 (+ 3371us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.111956 (+ 1829us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.112156 (+ 200us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:17.112441 (+ 285us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:17.120959 (+ 8518us) server_negotiation.cc:264] Negotiation successful 0627 19:31:17.121045 (+ 86us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":50285,"thread_start_us":48623,"threads_started":1} I0627 19:31:17.132354 19070 leader_election.cc:379] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [CANDIDATE]: Term 1 pre-election: Vote granted by peer 551f9818aeda4b50af2d9ac4e99b5380 (127.12.131.3:57511) I0627 19:31:17.132659 19070 leader_election.cc:270] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0627 19:31:17.132589 19232 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "551f9818aeda4b50af2d9ac4e99b5380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" is_pre_election: true I0627 19:31:17.133430 19232 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Snoozing failure detection for 1.698s (vote granted) I0627 19:31:17.133699 19232 raft_consensus.cc:2243] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 551f9818aeda4b50af2d9ac4e99b5380 in term 0. I0627 19:31:17.134400 19678 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Snoozing failure detection for 1.553s (election complete) I0627 19:31:17.134660 19678 raft_consensus.cc:2537] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0627 19:31:17.134878 19678 raft_consensus.cc:457] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0627 19:31:17.135102 19497 leader_election.cc:379] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) I0627 19:31:17.135121 19678 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Snoozing failure detection for 1.948s (starting election) I0627 19:31:17.135339 19497 leader_election.cc:270] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0627 19:31:17.135337 19678 raft_consensus.cc:2782] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:17.136014 19680 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Snoozing failure detection for 1.740s (election complete) I0627 19:31:17.136253 19680 raft_consensus.cc:2537] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0627 19:31:17.136502 19680 raft_consensus.cc:457] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0627 19:31:17.136713 19680 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Snoozing failure detection for 1.561s (starting election) I0627 19:31:17.136924 19680 raft_consensus.cc:2782] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:17.140971 19678 raft_consensus.cc:479] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:17.141948 19680 raft_consensus.cc:479] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:17.143121 19680 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 1 election: Requesting vote from peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974) I0627 19:31:17.143872 19680 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 1 election: Requesting vote from peer 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) I0627 19:31:17.145659 19232 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "551f9818aeda4b50af2d9ac4e99b5380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" I0627 19:31:17.147130 19232 raft_consensus.cc:2173] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 551f9818aeda4b50af2d9ac4e99b5380 in current term 1: Already voted for candidate 464e3b99b46c42f4a9d7c715d84bdf8e in this term. I0627 19:31:17.147814 19678 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [CANDIDATE]: Term 1 election: Requesting vote from peer 551f9818aeda4b50af2d9ac4e99b5380 (127.12.131.3:57511) I0627 19:31:17.148430 19678 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [CANDIDATE]: Term 1 election: Requesting vote from peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974) I0627 19:31:17.148705 19497 leader_election.cc:393] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 1 election: Vote denied by peer 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489). Message: Invalid argument: T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 551f9818aeda4b50af2d9ac4e99b5380 in current term 1: Already voted for candidate 464e3b99b46c42f4a9d7c715d84bdf8e in this term. I0627 19:31:17.149662 19582 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "551f9818aeda4b50af2d9ac4e99b5380" I0627 19:31:17.150297 19582 raft_consensus.cc:2173] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 464e3b99b46c42f4a9d7c715d84bdf8e in current term 1: Already voted for candidate 551f9818aeda4b50af2d9ac4e99b5380 in this term. I0627 19:31:17.151656 19070 leader_election.cc:393] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [CANDIDATE]: Term 1 election: Vote denied by peer 551f9818aeda4b50af2d9ac4e99b5380 (127.12.131.3:57511). Message: Invalid argument: T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 464e3b99b46c42f4a9d7c715d84bdf8e in current term 1: Already voted for candidate 551f9818aeda4b50af2d9ac4e99b5380 in this term. W0627 19:31:16.836151 19480 tablet.cc:1678] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:17.212024 19666 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.860245 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:46539 0627 19:31:16.860563 (+ 318us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:16.860588 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:17.199541 (+338953us) server_negotiation.cc:373] Connection header received 0627 19:31:17.199772 (+ 231us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:17.199800 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:17.199934 (+ 134us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:17.200050 (+ 116us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:31:17.201296 (+ 1246us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.201861 (+ 565us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.205729 (+ 3868us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.207435 (+ 1706us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.207630 (+ 195us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:17.207905 (+ 275us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:17.210799 (+ 2894us) server_negotiation.cc:264] Negotiation successful 0627 19:31:17.210881 (+ 82us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":94} I0627 19:31:17.215766 19686 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.859645 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.1:46489 0627 19:31:17.199365 (+339720us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:17.199437 (+ 72us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:17.199578 (+ 141us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:17.200374 (+ 796us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:17.200402 (+ 28us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:17.200486 (+ 84us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:31:17.200707 (+ 221us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.201049 (+ 342us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:17.201095 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.202207 (+ 1112us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.202235 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.204238 (+ 2003us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:17.204283 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.208275 (+ 3992us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.208303 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.208630 (+ 327us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:17.208775 (+ 145us) client_negotiation.cc:695] Sending connection context 0627 19:31:17.212943 (+ 4168us) client_negotiation.cc:225] Negotiation successful 0627 19:31:17.213012 (+ 69us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":339494,"thread_start_us":340015,"threads_started":1} I0627 19:31:17.226666 19685 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.857985 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.3:57511 0627 19:31:17.192949 (+334964us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:17.192999 (+ 50us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:17.193280 (+ 281us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:17.194114 (+ 834us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:17.194140 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:17.194193 (+ 53us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:31:17.194370 (+ 177us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.194574 (+ 204us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:17.194613 (+ 39us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.195600 (+ 987us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.195631 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.197443 (+ 1812us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:17.197488 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.222689 (+ 25201us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.222719 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.223036 (+ 317us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:17.223187 (+ 151us) client_negotiation.cc:695] Sending connection context 0627 19:31:17.225063 (+ 1876us) client_negotiation.cc:225] Negotiation successful 0627 19:31:17.225132 (+ 69us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":334722,"thread_start_us":339551,"threads_started":1} I0627 19:31:17.227025 19661 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:16.858829 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:45812 0627 19:31:16.859178 (+ 349us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:16.859201 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:17.193095 (+333894us) server_negotiation.cc:373] Connection header received 0627 19:31:17.193545 (+ 450us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:17.193572 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:17.193705 (+ 133us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:17.193821 (+ 116us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:31:17.194803 (+ 982us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.195323 (+ 520us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.220065 (+ 24742us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.221872 (+ 1807us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.222065 (+ 193us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:17.222335 (+ 270us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:17.223804 (+ 1469us) server_negotiation.cc:264] Negotiation successful 0627 19:31:17.223886 (+ 82us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":134} I0627 19:31:17.235816 19582 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "551f9818aeda4b50af2d9ac4e99b5380" is_pre_election: true I0627 19:31:17.236100 19702 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:17.052014 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.2:60974 0627 19:31:17.058153 (+ 6139us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:17.058227 (+ 74us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:17.058403 (+ 176us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:17.220709 (+162306us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:17.220736 (+ 27us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:17.220799 (+ 63us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:31:17.221104 (+ 305us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.221460 (+ 356us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:17.221502 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.224384 (+ 2882us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.224434 (+ 50us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.226601 (+ 2167us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:17.226646 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.233475 (+ 6829us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.233507 (+ 32us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.233861 (+ 354us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:17.234032 (+ 171us) client_negotiation.cc:695] Sending connection context 0627 19:31:17.234338 (+ 306us) client_negotiation.cc:225] Negotiation successful 0627 19:31:17.234437 (+ 99us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":3961,"mutex_wait_us":5,"thread_start_us":9699,"threads_started":1} I0627 19:31:17.236443 19582 raft_consensus.cc:2173] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 07e4d1e7f5c3453fb7f387b131ea8853 in current term 1: Already voted for candidate 551f9818aeda4b50af2d9ac4e99b5380 in this term. I0627 19:31:17.236572 19232 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" is_pre_election: true I0627 19:31:17.237156 19232 raft_consensus.cc:2173] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 07e4d1e7f5c3453fb7f387b131ea8853 in current term 1: Already voted for candidate 464e3b99b46c42f4a9d7c715d84bdf8e in this term. I0627 19:31:17.240521 19719 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:17.198032 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:46631 0627 19:31:17.220119 (+ 22087us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:17.220146 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:17.220184 (+ 38us) server_negotiation.cc:373] Connection header received 0627 19:31:17.220301 (+ 117us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:17.220326 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:17.220474 (+ 148us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:17.221673 (+ 1199us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:31:17.221939 (+ 266us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.222573 (+ 634us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.227755 (+ 5182us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.229568 (+ 1813us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.238170 (+ 8602us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:17.238486 (+ 316us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:17.239068 (+ 582us) server_negotiation.cc:264] Negotiation successful 0627 19:31:17.239151 (+ 83us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":195,"server-negotiator.queue_time_us":21876,"thread_start_us":21630,"threads_started":1} I0627 19:31:17.239393 19300 leader_election.cc:393] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 551f9818aeda4b50af2d9ac4e99b5380 (127.12.131.3:57511). Message: Invalid argument: T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 07e4d1e7f5c3453fb7f387b131ea8853 in current term 1: Already voted for candidate 551f9818aeda4b50af2d9ac4e99b5380 in this term. I0627 19:31:17.246647 19305 leader_election.cc:393] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489). Message: Invalid argument: T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 07e4d1e7f5c3453fb7f387b131ea8853 in current term 1: Already voted for candidate 464e3b99b46c42f4a9d7c715d84bdf8e in this term. I0627 19:31:17.247773 19305 leader_election.cc:270] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0627 19:31:17.248214 19704 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:17.071328 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.2:60974 0627 19:31:17.105531 (+ 34203us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:17.105624 (+ 93us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:17.105770 (+ 146us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:17.234271 (+128501us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:17.234301 (+ 30us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:17.234366 (+ 65us) client_negotiation.cc:175] Negotiated authn=CERTIFICATE 0627 19:31:17.234596 (+ 230us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.234917 (+ 321us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:17.234959 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.238528 (+ 3569us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.238554 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.240426 (+ 1872us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:17.240477 (+ 51us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.245952 (+ 5475us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.245982 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:17.246311 (+ 329us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:17.246493 (+ 182us) client_negotiation.cc:695] Sending connection context 0627 19:31:17.246751 (+ 258us) client_negotiation.cc:225] Negotiation successful 0627 19:31:17.246814 (+ 63us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":33935,"thread_start_us":29915,"threads_started":1} I0627 19:31:17.248729 19679 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Snoozing failure detection for 1.627s (election complete) I0627 19:31:17.250138 19679 raft_consensus.cc:2782] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:17.252377 19424 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "551f9818aeda4b50af2d9ac4e99b5380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" is_pre_election: true I0627 19:31:17.253967 19423 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "551f9818aeda4b50af2d9ac4e99b5380" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" I0627 19:31:17.254746 19720 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:17.200116 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:52589 0627 19:31:17.233732 (+ 33616us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:17.233760 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:17.233798 (+ 38us) server_negotiation.cc:373] Connection header received 0627 19:31:17.233908 (+ 110us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:17.233937 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:17.234062 (+ 125us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:17.234181 (+ 119us) server_negotiation.cc:197] Negotiated authn=CERTIFICATE 0627 19:31:17.235707 (+ 1526us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.236414 (+ 707us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.240706 (+ 4292us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:17.245665 (+ 4959us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:17.245871 (+ 206us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:17.246140 (+ 269us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:17.248623 (+ 2483us) server_negotiation.cc:264] Negotiation successful 0627 19:31:17.248696 (+ 73us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":33340,"thread_start_us":33124,"threads_started":1} I0627 19:31:17.255110 19679 raft_consensus.cc:2485] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0627 19:31:17.255949 19422 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" is_pre_election: true I0627 19:31:17.256331 19423 raft_consensus.cc:2218] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 551f9818aeda4b50af2d9ac4e99b5380 for term 1 because replica is already servicing an update from a current leader or another vote. I0627 19:31:17.256830 19422 raft_consensus.cc:2218] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 464e3b99b46c42f4a9d7c715d84bdf8e for term 1 because replica is already servicing an update from a current leader or another vote. I0627 19:31:17.256569 19421 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" I0627 19:31:17.257241 19421 raft_consensus.cc:2218] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 464e3b99b46c42f4a9d7c715d84bdf8e for term 1 because replica is already servicing an update from a current leader or another vote. I0627 19:31:17.257658 19424 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Snoozing failure detection for 1.767s (vote granted) I0627 19:31:17.257946 19424 raft_consensus.cc:2243] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 551f9818aeda4b50af2d9ac4e99b5380 in term 1. I0627 19:31:17.259565 19073 leader_election.cc:393] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [CANDIDATE]: Term 1 pre-election: Vote denied by peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974). Message: Service unavailable: T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 464e3b99b46c42f4a9d7c715d84bdf8e for term 1 because replica is already servicing an update from a current leader or another vote. I0627 19:31:17.260514 19073 leader_election.cc:393] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [CANDIDATE]: Term 1 election: Vote denied by peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974). Message: Service unavailable: T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 464e3b99b46c42f4a9d7c715d84bdf8e for term 1 because replica is already servicing an update from a current leader or another vote. I0627 19:31:17.260787 19073 leader_election.cc:270] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I0627 19:31:17.261483 19678 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Snoozing failure detection for 2.134s (election complete) I0627 19:31:17.261723 19678 raft_consensus.cc:2485] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0627 19:31:17.263434 19494 leader_election.cc:393] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 1 election: Vote denied by peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974). Message: Service unavailable: T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 551f9818aeda4b50af2d9ac4e99b5380 for term 1 because replica is already servicing an update from a current leader or another vote. I0627 19:31:17.263730 19494 leader_election.cc:270] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. I0627 19:31:17.264796 19494 leader_election.cc:379] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974) I0627 19:31:17.265393 19680 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Snoozing failure detection for 1.681s (election complete) I0627 19:31:17.265666 19680 raft_consensus.cc:2485] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0627 19:31:18.964784 19826 raft_consensus.cc:457] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0627 19:31:18.965332 19826 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Snoozing failure detection for 1.518s (starting election) I0627 19:31:18.965657 19826 raft_consensus.cc:479] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:18.966706 19826 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974) I0627 19:31:18.971616 19826 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) I0627 19:31:18.972307 19424 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "551f9818aeda4b50af2d9ac4e99b5380" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" is_pre_election: true I0627 19:31:18.972836 19424 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Snoozing failure detection for 1.722s (vote granted) I0627 19:31:18.973073 19424 raft_consensus.cc:2243] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 551f9818aeda4b50af2d9ac4e99b5380 in term 1. I0627 19:31:18.973461 19232 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "551f9818aeda4b50af2d9ac4e99b5380" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" is_pre_election: true I0627 19:31:18.973951 19232 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Snoozing failure detection for 1.982s (vote granted) I0627 19:31:18.974191 19232 raft_consensus.cc:2243] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 551f9818aeda4b50af2d9ac4e99b5380 in term 1. I0627 19:31:18.975600 19494 leader_election.cc:379] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974) I0627 19:31:18.975863 19494 leader_election.cc:270] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0627 19:31:18.976153 19497 leader_election.cc:379] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) I0627 19:31:18.976495 19826 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Snoozing failure detection for 2.684s (election complete) I0627 19:31:18.976789 19826 raft_consensus.cc:2537] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0627 19:31:18.977021 19826 raft_consensus.cc:457] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0627 19:31:18.977252 19826 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Snoozing failure detection for 1.936s (starting election) I0627 19:31:18.977514 19826 raft_consensus.cc:2782] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 1 FOLLOWER]: Advancing to term 2 I0627 19:31:18.981861 19826 raft_consensus.cc:479] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:18.987053 19826 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 2 election: Requesting vote from peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974) I0627 19:31:18.988216 19826 leader_election.cc:243] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 2 election: Requesting vote from peer 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) I0627 19:31:18.989235 19424 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "551f9818aeda4b50af2d9ac4e99b5380" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" I0627 19:31:18.989692 19424 raft_consensus.cc:2782] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 1 FOLLOWER]: Advancing to term 2 I0627 19:31:18.989959 19424 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Snoozing failure detection for 2.071s (vote granted) I0627 19:31:18.990352 19232 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "32014c9a31f14216874c40a94fc74717" candidate_uuid: "551f9818aeda4b50af2d9ac4e99b5380" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" I0627 19:31:18.990809 19232 raft_consensus.cc:2782] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 1 FOLLOWER]: Advancing to term 2 I0627 19:31:18.991035 19232 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e: Snoozing failure detection for 2.834s (vote granted) I0627 19:31:18.994956 19424 raft_consensus.cc:2243] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 551f9818aeda4b50af2d9ac4e99b5380 in term 2. I0627 19:31:18.996215 19494 leader_election.cc:379] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 2 election: Vote granted by peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974) I0627 19:31:18.996207 19232 raft_consensus.cc:2243] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 551f9818aeda4b50af2d9ac4e99b5380 in term 2. I0627 19:31:18.996505 19494 leader_election.cc:270] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0627 19:31:18.997795 19497 leader_election.cc:379] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [CANDIDATE]: Term 2 election: Vote granted by peer 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) I0627 19:31:18.998356 19826 raft_consensus.cc:2730] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Snoozing failure detection for 2.936s (election complete) I0627 19:31:18.998623 19826 raft_consensus.cc:2537] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 2 FOLLOWER]: Leader election won for term 2 I0627 19:31:19.000175 19826 raft_consensus.cc:593] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 2 LEADER]: Becoming Leader. State: Replica: 551f9818aeda4b50af2d9ac4e99b5380, State: Running, Role: LEADER I0627 19:31:19.000785 19826 consensus_queue.cc:204] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } } I0627 19:31:19.015408 18973 catalog_manager.cc:3798] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 reported cstate change: term changed from 0 to 2, leader changed from to 551f9818aeda4b50af2d9ac4e99b5380 (127.12.131.3). New cstate: current_term: 2 leader_uuid: "551f9818aeda4b50af2d9ac4e99b5380" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551f9818aeda4b50af2d9ac4e99b5380" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 } health_report { overall_health: UNKNOWN } } } I0627 19:31:19.116999 19847 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:19.068530 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:34902 0627 19:31:19.098838 (+ 30308us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:19.098866 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:19.098911 (+ 45us) server_negotiation.cc:373] Connection header received 0627 19:31:19.099079 (+ 168us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:19.099106 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:19.099286 (+ 180us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:19.099471 (+ 185us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:31:19.106004 (+ 6533us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:19.106602 (+ 598us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:19.109399 (+ 2797us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:19.110557 (+ 1158us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:19.110754 (+ 197us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:19.114465 (+ 3711us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:31:19.114962 (+ 497us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:31:19.115114 (+ 152us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:19.115437 (+ 323us) server_negotiation.cc:264] Negotiation successful 0627 19:31:19.115632 (+ 195us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":30076,"thread_start_us":18943,"threads_started":1} I0627 19:31:19.155951 19424 raft_consensus.cc:1081] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 2 FOLLOWER]: Refusing update from remote peer 551f9818aeda4b50af2d9ac4e99b5380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0627 19:31:19.156002 19232 raft_consensus.cc:1081] T 32014c9a31f14216874c40a94fc74717 P 464e3b99b46c42f4a9d7c715d84bdf8e [term 2 FOLLOWER]: Refusing update from remote peer 551f9818aeda4b50af2d9ac4e99b5380: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0627 19:31:19.157738 19826 consensus_queue.cc:962] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464e3b99b46c42f4a9d7c715d84bdf8e" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 46489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0627 19:31:19.160352 19826 consensus_queue.cc:962] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07e4d1e7f5c3453fb7f387b131ea8853" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60974 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0627 19:31:19.239286 18973 catalog_manager.cc:1731] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.12.131.62:53847: table { table_name: "test-table" } I0627 19:31:19.246973 18972 catalog_manager.cc:3675] Got report from deleted tablet 32014c9a31f14216874c40a94fc74717 (table test-table [id=dc0c540c90bf46f4853978eba9ea3ded]) (Table deleted at 2018-06-27 19:31:19 PDT): Sending delete request for this tablet I0627 19:31:19.247460 18973 catalog_manager.cc:4037] T 00000000000000000000000000000000 P cb35f880a5c14c5a8a94dcc34bef3a17: Sending DeleteTablet for 3 replicas of tablet 32014c9a31f14216874c40a94fc74717 I0627 19:31:19.247774 18973 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 32014c9a31f14216874c40a94fc74717 on 551f9818aeda4b50af2d9ac4e99b5380 (127.12.131.3:57511) (Table deleted at 2018-06-27 19:31:19 PDT) I0627 19:31:19.248270 18973 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 32014c9a31f14216874c40a94fc74717 on 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974) (Table deleted at 2018-06-27 19:31:19 PDT) I0627 19:31:19.248688 18973 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 32014c9a31f14216874c40a94fc74717 on 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) (Table deleted at 2018-06-27 19:31:19 PDT) I0627 19:31:19.247772 18972 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 32014c9a31f14216874c40a94fc74717 on 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) (Table deleted at 2018-06-27 19:31:19 PDT) I0627 19:31:19.250519 19555 tablet_service.cc:800] Processing DeleteTablet for tablet 32014c9a31f14216874c40a94fc74717 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-27 19:31:19 PDT) from {username='kudu', principal='kudu/127.12.131.62@KRBTEST.COM'} at 127.12.131.3:34899 I0627 19:31:19.251299 18973 catalog_manager.cc:3675] Got report from deleted tablet 32014c9a31f14216874c40a94fc74717 (table test-table [id=dc0c540c90bf46f4853978eba9ea3ded]) (Table deleted at 2018-06-27 19:31:19 PDT): Sending delete request for this tablet I0627 19:31:19.252607 19388 tablet_service.cc:800] Processing DeleteTablet for tablet 32014c9a31f14216874c40a94fc74717 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-27 19:31:19 PDT) from {username='kudu', principal='kudu/127.12.131.62@KRBTEST.COM'} at 127.12.131.2:48175 I0627 19:31:19.252745 19209 tablet_service.cc:800] Processing DeleteTablet for tablet 32014c9a31f14216874c40a94fc74717 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-27 19:31:19 PDT) from {username='kudu', principal='kudu/127.12.131.62@KRBTEST.COM'} at 127.12.131.1:41986 I0627 19:31:19.253463 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 19016 I0627 19:31:19.254094 19210 tablet_service.cc:800] Processing DeleteTablet for tablet 32014c9a31f14216874c40a94fc74717 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-27 19:31:19 PDT) from {username='kudu', principal='kudu/127.12.131.62@KRBTEST.COM'} at 127.12.131.1:41986 I0627 19:31:19.255300 18973 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 32014c9a31f14216874c40a94fc74717 on 551f9818aeda4b50af2d9ac4e99b5380 (127.12.131.3:57511) (Table deleted at 2018-06-27 19:31:19 PDT) I0627 19:31:19.256752 19554 tablet_service.cc:800] Processing DeleteTablet for tablet 32014c9a31f14216874c40a94fc74717 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-27 19:31:19 PDT) from {username='kudu', principal='kudu/127.12.131.62@KRBTEST.COM'} at 127.12.131.3:34899 I0627 19:31:19.256045 19882 tablet_replica.cc:261] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: stopping tablet replica I0627 19:31:19.262910 19882 maintenance_manager.cc:237] P 551f9818aeda4b50af2d9ac4e99b5380: Unregistered op CompactRowSetsOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.263463 19882 maintenance_manager.cc:237] P 551f9818aeda4b50af2d9ac4e99b5380: Unregistered op MinorDeltaCompactionOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.263926 19882 maintenance_manager.cc:237] P 551f9818aeda4b50af2d9ac4e99b5380: Unregistered op MajorDeltaCompactionOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.264354 19882 maintenance_manager.cc:237] P 551f9818aeda4b50af2d9ac4e99b5380: Unregistered op UndoDeltaBlockGCOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.264045 19883 tablet_replica.cc:261] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: stopping tablet replica I0627 19:31:19.264840 19883 maintenance_manager.cc:237] P 07e4d1e7f5c3453fb7f387b131ea8853: Unregistered op CompactRowSetsOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.264868 19882 maintenance_manager.cc:237] P 551f9818aeda4b50af2d9ac4e99b5380: Unregistered op FlushMRSOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.265060 19882 maintenance_manager.cc:237] P 551f9818aeda4b50af2d9ac4e99b5380: Unregistered op FlushDeltaMemStoresOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.265240 19882 maintenance_manager.cc:237] P 551f9818aeda4b50af2d9ac4e99b5380: Unregistered op LogGCOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.265473 19882 raft_consensus.cc:2040] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 2 LEADER]: Raft consensus shutting down. I0627 19:31:19.265305 19883 maintenance_manager.cc:237] P 07e4d1e7f5c3453fb7f387b131ea8853: Unregistered op MinorDeltaCompactionOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.265697 19882 consensus_peers.cc:455] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 -> Peer 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489): Closing peer: 464e3b99b46c42f4a9d7c715d84bdf8e I0627 19:31:19.265928 19882 consensus_peers.cc:455] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 -> Peer 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974): Closing peer: 07e4d1e7f5c3453fb7f387b131ea8853 I0627 19:31:19.266120 19883 maintenance_manager.cc:237] P 07e4d1e7f5c3453fb7f387b131ea8853: Unregistered op MajorDeltaCompactionOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.266342 19883 maintenance_manager.cc:237] P 07e4d1e7f5c3453fb7f387b131ea8853: Unregistered op UndoDeltaBlockGCOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.266405 19882 raft_consensus.cc:2067] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380 [term 2 FOLLOWER]: Raft consensus is shut down! I0627 19:31:19.268388 19883 maintenance_manager.cc:237] P 07e4d1e7f5c3453fb7f387b131ea8853: Unregistered op FlushMRSOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.268637 19883 maintenance_manager.cc:237] P 07e4d1e7f5c3453fb7f387b131ea8853: Unregistered op FlushDeltaMemStoresOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.268843 19883 maintenance_manager.cc:237] P 07e4d1e7f5c3453fb7f387b131ea8853: Unregistered op LogGCOp(32014c9a31f14216874c40a94fc74717) I0627 19:31:19.269076 19883 raft_consensus.cc:2040] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 2 FOLLOWER]: Raft consensus shutting down. I0627 19:31:19.269409 19883 raft_consensus.cc:2067] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853 [term 2 FOLLOWER]: Raft consensus is shut down! I0627 19:31:19.276046 19883 ts_tablet_manager.cc:1369] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Deleting tablet data with delete state TABLET_DATA_DELETED I0627 19:31:19.280025 19882 ts_tablet_manager.cc:1369] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Deleting tablet data with delete state TABLET_DATA_DELETED I0627 19:31:19.304915 19882 ts_tablet_manager.cc:1382] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2 I0627 19:31:19.311470 19882 log.cc:986] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-2/wal/wals/32014c9a31f14216874c40a94fc74717 I0627 19:31:19.312292 19882 ts_tablet_manager.cc:1403] T 32014c9a31f14216874c40a94fc74717 P 551f9818aeda4b50af2d9ac4e99b5380: Deleting consensus metadata I0627 19:31:19.312351 19883 ts_tablet_manager.cc:1382] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2 I0627 19:31:19.322106 19883 log.cc:986] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableWebUI.1530153042096250-12812/minicluster-data/ts-1/wal/wals/32014c9a31f14216874c40a94fc74717 I0627 19:31:19.322970 19883 ts_tablet_manager.cc:1403] T 32014c9a31f14216874c40a94fc74717 P 07e4d1e7f5c3453fb7f387b131ea8853: Deleting consensus metadata I0627 19:31:19.324249 18903 catalog_manager.cc:3103] TS 551f9818aeda4b50af2d9ac4e99b5380 (127.12.131.3:57511): tablet 32014c9a31f14216874c40a94fc74717 (table test-table [id=dc0c540c90bf46f4853978eba9ea3ded]) successfully deleted W0627 19:31:19.325000 18903 catalog_manager.cc:3083] TS 551f9818aeda4b50af2d9ac4e99b5380 (127.12.131.3:57511): delete failed for tablet 32014c9a31f14216874c40a94fc74717 because the tablet was not found. No further retry: Not found: Tablet not found: 32014c9a31f14216874c40a94fc74717 I0627 19:31:19.325691 18905 catalog_manager.cc:3103] TS 07e4d1e7f5c3453fb7f387b131ea8853 (127.12.131.2:60974): tablet 32014c9a31f14216874c40a94fc74717 (table test-table [id=dc0c540c90bf46f4853978eba9ea3ded]) successfully deleted W0627 19:31:19.341002 18905 connection.cc:512] server connection from 127.12.131.1:55021 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:55021): Connection reset by peer (error 104) W0627 19:31:19.340945 18924 catalog_manager.cc:2867] TS 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489): Delete Tablet RPC failed for tablet 32014c9a31f14216874c40a94fc74717: Network error: recv got EOF from 127.12.131.1:46489 (error 108) I0627 19:31:19.341326 18924 catalog_manager.cc:2903] Scheduling retry of 32014c9a31f14216874c40a94fc74717 Delete Tablet RPC for TS=464e3b99b46c42f4a9d7c715d84bdf8e with a delay of 32 ms (attempt = 1) W0627 19:31:19.341642 18924 catalog_manager.cc:2867] TS 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489): Delete Tablet RPC failed for tablet 32014c9a31f14216874c40a94fc74717: Network error: recv got EOF from 127.12.131.1:46489 (error 108) I0627 19:31:19.341840 18924 catalog_manager.cc:2903] Scheduling retry of 32014c9a31f14216874c40a94fc74717 Delete Tablet RPC for TS=464e3b99b46c42f4a9d7c715d84bdf8e with a delay of 63 ms (attempt = 1) W0627 19:31:19.342068 19305 connection.cc:512] client connection to 127.12.131.1:46489 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:46489): Connection reset by peer (error 104) W0627 19:31:19.343142 19497 connection.cc:512] client connection to 127.12.131.1:46489 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:46489): Connection reset by peer (error 104) W0627 19:31:19.344362 19493 connection.cc:512] server connection from 127.12.131.1:43381 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:43381): Connection reset by peer (error 104) W0627 19:31:19.345157 19305 connection.cc:512] server connection from 127.12.131.1:52589 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:52589): Connection reset by peer (error 104) I0627 19:31:19.346551 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 19280 I0627 19:31:19.374882 18924 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 32014c9a31f14216874c40a94fc74717 on 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) (Table deleted at 2018-06-27 19:31:19 PDT) I0627 19:31:19.391399 19898 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:19.375568 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.1:46489 0627 19:31:19.383817 (+ 8249us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:19.384015 (+ 198us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.1:46489: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":8014,"threads_started":1} W0627 19:31:19.391762 18924 catalog_manager.cc:2867] TS 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489): Delete Tablet RPC failed for tablet 32014c9a31f14216874c40a94fc74717: Network error: Client connection negotiation failed: client connection to 127.12.131.1:46489: connect: Connection refused (error 111) I0627 19:31:19.392015 18924 catalog_manager.cc:2903] Scheduling retry of 32014c9a31f14216874c40a94fc74717 Delete Tablet RPC for TS=464e3b99b46c42f4a9d7c715d84bdf8e with a delay of 80 ms (attempt = 2) W0627 19:31:19.392081 18903 connection.cc:512] server connection from 127.12.131.2:52315 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:52315): Connection reset by peer (error 104) W0627 19:31:19.392115 19497 connection.cc:512] server connection from 127.12.131.2:45812 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:45812): Connection reset by peer (error 104) W0627 19:31:19.392607 19494 connection.cc:512] client connection to 127.12.131.2:60974 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:60974): Connection reset by peer (error 104) I0627 19:31:19.393018 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 19484 I0627 19:31:19.405916 18924 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 32014c9a31f14216874c40a94fc74717 on 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489) (Table deleted at 2018-06-27 19:31:19 PDT) I0627 19:31:19.409386 19898 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:19.406547 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.1:46489 0627 19:31:19.408784 (+ 2237us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:19.408944 (+ 160us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.1:46489: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":2018} W0627 19:31:19.409837 18924 catalog_manager.cc:2867] TS 464e3b99b46c42f4a9d7c715d84bdf8e (127.12.131.1:46489): Delete Tablet RPC failed for tablet 32014c9a31f14216874c40a94fc74717: Network error: Client connection negotiation failed: client connection to 127.12.131.1:46489: connect: Connection refused (error 111) I0627 19:31:19.410115 18924 catalog_manager.cc:2903] Scheduling retry of 32014c9a31f14216874c40a94fc74717 Delete Tablet RPC for TS=464e3b99b46c42f4a9d7c715d84bdf8e with a delay of 79 ms (attempt = 2) W0627 19:31:19.462880 18903 connection.cc:512] server connection from 127.12.131.3:42028 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.3:42028): Connection reset by peer (error 104) I0627 19:31:19.462954 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 18830 [ OK ] SecurityITest.TestDisableWebUI (9657 ms) [ RUN ] SecurityITest.TestDisableAuthenticationEncryption I0627 19:31:19.536391 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:52835 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_trace_negotiation --rpc_authentication=disabled --rpc_encryption=disabled --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:19.803522 19916 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:19.809876 19916 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:19.841706 19916 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:19.841991 19916 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:19.842191 19916 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:19.842367 19916 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:19.842594 19916 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0627 19:31:19.887228 19916 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:52835 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --rpc_authentication=disabled --rpc_encryption=disabled --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:19.902837 19916 master_main.cc:80] Initializing master server... I0627 19:31:19.906504 19916 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 208592us I0627 19:31:19.907654 19916 fs_manager.cc:261] Metadata directory not provided I0627 19:31:19.907881 19916 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:19.908277 19916 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:19.908504 19916 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:19.924376 19916 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" format_stamp: "Formatted at 2018-06-28 02:31:19 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:19.925566 19916 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" format_stamp: "Formatted at 2018-06-28 02:31:19 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:19.963704 19916 fs_manager.cc:501] Time spent creating directory manager: real 0.038s user 0.013s sys 0.013s I0627 19:31:19.965561 19916 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:19.966950 19916 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:19.984766 19916 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.002s sys 0.004s I0627 19:31:19.985050 19916 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/wal uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" format_stamp: "Formatted at 2018-06-28 02:31:19 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:19.985505 19916 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:20.346940 19916 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:20.355159 19916 master_main.cc:83] Starting Master server... I0627 19:31:20.683030 19916 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:52835 I0627 19:31:20.684470 19916 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:31:20.684677 19916 webserver.cc:186] Document root disabled I0627 19:31:20.691399 19916 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:55079/ I0627 19:31:20.692646 19916 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:20.701140 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 19916 I0627 19:31:20.702590 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.12.131.1:0 --local_ip_for_outbound_sockets=127.12.131.1 --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:52835 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_trace_negotiation --rpc_authentication=disabled --rpc_encryption=disabled with env {} I0627 19:31:20.704551 20108 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:20.730216 20108 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf: Bootstrap starting. I0627 19:31:20.735152 20108 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf: No blocks or log segments found. Creating new log. I0627 19:31:20.736866 20108 log.cc:525] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf: Log is configured to *not* fsync() on all Append() calls I0627 19:31:20.744807 20108 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf: No bootstrap required, opened a new log I0627 19:31:20.766571 20108 raft_consensus.cc:325] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" member_type: VOTER } I0627 19:31:20.767185 20108 raft_consensus.cc:351] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:20.767463 20108 raft_consensus.cc:628] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0d2724a2fb42f7bb0f41b0251080bf, State: Initialized, Role: FOLLOWER I0627 19:31:20.768489 20108 consensus_queue.cc:227] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" member_type: VOTER } I0627 19:31:20.769049 20108 raft_consensus.cc:365] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:20.769335 20108 raft_consensus.cc:457] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:20.769709 20108 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf: Snoozing failure detection for 2.059s (starting election) I0627 19:31:20.769985 20108 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:20.774636 20108 raft_consensus.cc:479] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" member_type: VOTER } I0627 19:31:20.775435 20108 leader_election.cc:270] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:20.781589 20120 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf: Snoozing failure detection for 2.010s (election complete) I0627 19:31:20.782927 20120 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:20.788246 20120 raft_consensus.cc:593] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [term 1 LEADER]: Becoming Leader. State: Replica: 6c0d2724a2fb42f7bb0f41b0251080bf, State: Running, Role: LEADER I0627 19:31:20.789134 20120 consensus_queue.cc:204] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" member_type: VOTER } I0627 19:31:20.800709 20108 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" member_type: VOTER } } I0627 19:31:20.801159 20108 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [sys.catalog]: This master's current role is: LEADER I0627 19:31:20.802703 20122 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" member_type: VOTER } } I0627 19:31:21.029747 20122 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [sys.catalog]: This master's current role is: LEADER I0627 19:31:20.806711 20121 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c0d2724a2fb42f7bb0f41b0251080bf. Latest consensus state: current_term: 1 leader_uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c0d2724a2fb42f7bb0f41b0251080bf" member_type: VOTER } } I0627 19:31:21.037998 20121 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [sys.catalog]: This master's current role is: LEADER I0627 19:31:21.039613 20108 sys_catalog.cc:424] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:21.039868 20124 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:21.047125 20145 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0627 19:31:21.050544 19916 master_main.cc:86] Master server successfully started. I0627 19:31:21.060155 20124 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:21.070864 20111 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:21.071646 20111 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:21.091328 20111 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:21.091645 20111 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:21.092284 20111 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.1 W0627 19:31:21.112483 20015 tablet.cc:1678] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:21.168567 20111 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.1:0 --rpc_server_allow_ephemeral_ports=true --rpc_authentication=disabled --rpc_encryption=disabled --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:52835 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:21.184597 20111 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:21.187729 20111 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 209592us I0627 19:31:21.188575 20111 fs_manager.cc:261] Metadata directory not provided I0627 19:31:21.188822 20111 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:21.189256 20111 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:21.189489 20111 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:21.205576 20111 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/data/instance: uuid: "3e7bb63625d4482fb8e1a864a845255d" format_stamp: "Formatted at 2018-06-28 02:31:21 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:21.206742 20111 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/wal/instance: uuid: "3e7bb63625d4482fb8e1a864a845255d" format_stamp: "Formatted at 2018-06-28 02:31:21 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:21.214681 20124 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:21.215184 20124 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:21.233698 20111 fs_manager.cc:501] Time spent creating directory manager: real 0.026s user 0.014s sys 0.010s I0627 19:31:21.235581 20111 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:21.236238 20111 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:21.257982 20124 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf: Generated new TSK 0 I0627 19:31:21.263166 20111 fs_manager.cc:417] Time spent opening block manager: real 0.016s user 0.000s sys 0.012s I0627 19:31:21.263429 20111 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/wal uuid: "3e7bb63625d4482fb8e1a864a845255d" format_stamp: "Formatted at 2018-06-28 02:31:21 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:21.263857 20111 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:21.618849 20111 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:21.629964 20111 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:21.641479 20111 tablet_server_main.cc:90] Starting tablet server... I0627 19:31:22.778272 20111 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.1:33551 I0627 19:31:22.786607 20111 webserver.cc:175] Starting webserver on 127.12.131.1:0 I0627 19:31:22.786785 20111 webserver.cc:186] Document root disabled I0627 19:31:22.793300 20111 webserver.cc:313] Webserver started. Bound to: http://127.12.131.1:59543/ I0627 19:31:22.794371 20111 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-0/data/info.pb I0627 19:31:22.797857 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 20111 I0627 19:31:22.799242 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.12.131.2:0 --local_ip_for_outbound_sockets=127.12.131.2 --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:52835 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_trace_negotiation --rpc_authentication=disabled --rpc_encryption=disabled with env {} I0627 19:31:22.827041 20111 tablet_server_main.cc:93] Tablet server successfully started. I0627 19:31:22.857285 20274 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:22.810615 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:58789 0627 19:31:22.816887 (+ 6272us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:22.816923 (+ 36us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:22.826375 (+ 9452us) server_negotiation.cc:373] Connection header received 0627 19:31:22.850559 (+ 24184us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:22.850604 (+ 45us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:22.850807 (+ 203us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:22.851138 (+ 331us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:22.854580 (+ 3442us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:22.854613 (+ 33us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:22.854640 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:22.854691 (+ 51us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:22.854804 (+ 113us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:22.854862 (+ 58us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:22.854985 (+ 123us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:22.855716 (+ 731us) server_negotiation.cc:264] Negotiation successful 0627 19:31:22.855888 (+ 172us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5884,"thread_start_us":5404,"threads_started":1} I0627 19:31:22.865774 20275 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:22.810647 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:52835 0627 19:31:22.825009 (+ 14362us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:22.825119 (+ 110us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:22.849911 (+ 24792us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:22.852200 (+ 2289us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:22.852231 (+ 31us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:22.852350 (+ 119us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:22.852520 (+ 170us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0627 19:31:22.852541 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:22.852594 (+ 53us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0627 19:31:22.852742 (+ 148us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:22.855132 (+ 2390us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:22.855162 (+ 30us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:22.855190 (+ 28us) client_negotiation.cc:695] Sending connection context 0627 19:31:22.863935 (+ 8745us) client_negotiation.cc:225] Negotiation successful 0627 19:31:22.864173 (+ 238us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":13588,"thread_start_us":5467,"threads_started":1} I0627 19:31:22.868371 20271 heartbeater.cc:346] Connected to a master server at 127.12.131.62:52835 I0627 19:31:22.868726 20271 heartbeater.cc:426] Registering TS with master... I0627 19:31:22.869273 20271 heartbeater.cc:475] Master 127.12.131.62:52835 requested a full tablet report, sending... I0627 19:31:22.871357 20037 ts_manager.cc:98] Registered new tserver with Master: 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551) WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:23.148087 20272 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:23.151926 20272 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:23.180480 20272 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:23.188398 20272 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:23.189077 20272 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.2 I0627 19:31:23.262465 20272 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.2:0 --rpc_server_allow_ephemeral_ports=true --rpc_authentication=disabled --rpc_encryption=disabled --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:52835 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.2 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:23.287978 20272 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:23.293596 20272 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 210592us I0627 19:31:23.296015 20272 fs_manager.cc:261] Metadata directory not provided I0627 19:31:23.296695 20272 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:23.297145 20272 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:23.297359 20272 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:23.313421 20272 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/data/instance: uuid: "87ef418b73d04687969fbd353b1085cd" format_stamp: "Formatted at 2018-06-28 02:31:23 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:23.314545 20272 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/wal/instance: uuid: "87ef418b73d04687969fbd353b1085cd" format_stamp: "Formatted at 2018-06-28 02:31:23 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:23.374272 20272 fs_manager.cc:501] Time spent creating directory manager: real 0.059s user 0.014s sys 0.010s I0627 19:31:23.376329 20272 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:23.377029 20272 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:23.401331 20272 fs_manager.cc:417] Time spent opening block manager: real 0.014s user 0.002s sys 0.004s I0627 19:31:23.401603 20272 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/wal uuid: "87ef418b73d04687969fbd353b1085cd" format_stamp: "Formatted at 2018-06-28 02:31:23 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:23.401993 20272 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:23.769039 20272 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:23.778589 20272 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:23.785751 20272 tablet_server_main.cc:90] Starting tablet server... I0627 19:31:23.873764 20271 heartbeater.cc:467] Master 127.12.131.62:52835 was elected leader, sending a full tablet report... I0627 19:31:24.459779 20272 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.2:60127 I0627 19:31:24.461586 20272 webserver.cc:175] Starting webserver on 127.12.131.2:0 I0627 19:31:24.461750 20272 webserver.cc:186] Document root disabled I0627 19:31:24.463703 20272 webserver.cc:313] Webserver started. Bound to: http://127.12.131.2:37072/ I0627 19:31:24.464718 20272 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/data/info.pb I0627 19:31:24.472462 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 20272 I0627 19:31:24.473867 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.12.131.3:0 --local_ip_for_outbound_sockets=127.12.131.3 --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:52835 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_trace_negotiation --rpc_authentication=disabled --rpc_encryption=disabled with env {} I0627 19:31:24.474819 20272 tablet_server_main.cc:93] Tablet server successfully started. I0627 19:31:24.500048 20401 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:24.469715 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:33536 0627 19:31:24.475102 (+ 5387us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:24.475127 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:24.477111 (+ 1984us) server_negotiation.cc:373] Connection header received 0627 19:31:24.496231 (+ 19120us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:24.496264 (+ 33us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:24.496408 (+ 144us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:24.496553 (+ 145us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:24.497796 (+ 1243us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:24.497823 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:24.497846 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:24.497902 (+ 56us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:24.497964 (+ 62us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:24.498020 (+ 56us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:24.498117 (+ 97us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:24.498653 (+ 536us) server_negotiation.cc:264] Negotiation successful 0627 19:31:24.498747 (+ 94us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5182,"thread_start_us":9655,"threads_started":1} I0627 19:31:24.503087 20400 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:24.468360 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:52835 0627 19:31:24.476130 (+ 7770us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:24.476203 (+ 73us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:24.495874 (+ 19671us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:24.497001 (+ 1127us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:24.497033 (+ 32us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:24.497151 (+ 118us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:24.497286 (+ 135us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0627 19:31:24.497308 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:24.497376 (+ 68us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0627 19:31:24.497543 (+ 167us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:24.498264 (+ 721us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:24.498293 (+ 29us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:24.498320 (+ 27us) client_negotiation.cc:695] Sending connection context 0627 19:31:24.501406 (+ 3086us) client_negotiation.cc:225] Negotiation successful 0627 19:31:24.501596 (+ 190us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":7226,"thread_start_us":8041,"threads_started":1} I0627 19:31:24.506392 20398 heartbeater.cc:346] Connected to a master server at 127.12.131.62:52835 I0627 19:31:24.506745 20398 heartbeater.cc:426] Registering TS with master... I0627 19:31:24.507285 20398 heartbeater.cc:475] Master 127.12.131.62:52835 requested a full tablet report, sending... I0627 19:31:24.508826 20037 ts_manager.cc:98] Registered new tserver with Master: 87ef418b73d04687969fbd353b1085cd (127.12.131.2:60127) WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:24.689994 20402 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:24.690704 20402 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:24.708091 20402 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:24.708339 20402 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:24.708986 20402 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.3 I0627 19:31:24.741940 20402 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.3:0 --rpc_server_allow_ephemeral_ports=true --rpc_authentication=disabled --rpc_encryption=disabled --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:52835 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.3 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:24.757172 20402 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:24.760107 20402 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 211092us I0627 19:31:24.760864 20402 fs_manager.cc:261] Metadata directory not provided I0627 19:31:24.761073 20402 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:24.761500 20402 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:24.761708 20402 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:24.778350 20402 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/data/instance: uuid: "0090108d1d7d43868dae9f524043684e" format_stamp: "Formatted at 2018-06-28 02:31:24 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:24.779546 20402 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/wal/instance: uuid: "0090108d1d7d43868dae9f524043684e" format_stamp: "Formatted at 2018-06-28 02:31:24 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:24.818075 20402 fs_manager.cc:501] Time spent creating directory manager: real 0.038s user 0.014s sys 0.005s I0627 19:31:24.820026 20402 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:24.820694 20402 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:24.849768 20402 fs_manager.cc:417] Time spent opening block manager: real 0.009s user 0.003s sys 0.003s I0627 19:31:24.850014 20402 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/wal uuid: "0090108d1d7d43868dae9f524043684e" format_stamp: "Formatted at 2018-06-28 02:31:24 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:24.850383 20402 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:25.047158 20402 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:25.052289 20402 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:25.053900 20402 tablet_server_main.cc:90] Starting tablet server... I0627 19:31:25.404049 20402 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.3:57976 I0627 19:31:25.405879 20402 webserver.cc:175] Starting webserver on 127.12.131.3:0 I0627 19:31:25.406065 20402 webserver.cc:186] Document root disabled I0627 19:31:25.408233 20402 webserver.cc:313] Webserver started. Bound to: http://127.12.131.3:48407/ I0627 19:31:25.409354 20402 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/data/info.pb I0627 19:31:25.417912 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 20402 I0627 19:31:25.420959 20402 tablet_server_main.cc:93] Tablet server successfully started. I0627 19:31:25.436864 20529 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.420112 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:44305 0627 19:31:25.431155 (+ 11043us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:25.431182 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:25.431222 (+ 40us) server_negotiation.cc:373] Connection header received 0627 19:31:25.431342 (+ 120us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:25.431424 (+ 82us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:25.431549 (+ 125us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:25.431699 (+ 150us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:25.432630 (+ 931us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:25.432658 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:25.432681 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:25.432726 (+ 45us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:25.432788 (+ 62us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:25.432841 (+ 53us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:25.432950 (+ 109us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:25.435424 (+ 2474us) server_negotiation.cc:264] Negotiation successful 0627 19:31:25.435513 (+ 89us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":10825,"thread_start_us":1694,"threads_started":1} I0627 19:31:25.449245 20531 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.416484 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:36463 0627 19:31:25.422273 (+ 5789us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:25.422298 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:25.423478 (+ 1180us) server_negotiation.cc:373] Connection header received 0627 19:31:25.444485 (+ 21007us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:25.444514 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:25.444626 (+ 112us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:25.444768 (+ 142us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:25.446009 (+ 1241us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:25.446038 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:25.446061 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:25.446114 (+ 53us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:25.446174 (+ 60us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:25.446226 (+ 52us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:25.446327 (+ 101us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:25.447254 (+ 927us) server_negotiation.cc:264] Negotiation successful 0627 19:31:25.447345 (+ 91us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5585,"thread_start_us":9650,"threads_started":1} I0627 19:31:25.449911 20528 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.414885 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:52835 0627 19:31:25.423224 (+ 8339us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:25.423334 (+ 110us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:25.443281 (+ 19947us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:25.445030 (+ 1749us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:25.445063 (+ 33us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:25.445190 (+ 127us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:25.445332 (+ 142us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0627 19:31:25.445354 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:25.445432 (+ 78us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0627 19:31:25.445606 (+ 174us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:25.446892 (+ 1286us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:25.446922 (+ 30us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:25.446950 (+ 28us) client_negotiation.cc:695] Sending connection context 0627 19:31:25.447122 (+ 172us) client_negotiation.cc:225] Negotiation successful 0627 19:31:25.447337 (+ 215us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":7413,"thread_start_us":6853,"threads_started":1} I0627 19:31:25.452054 20526 heartbeater.cc:346] Connected to a master server at 127.12.131.62:52835 I0627 19:31:25.452368 20526 heartbeater.cc:426] Registering TS with master... I0627 19:31:25.452919 20526 heartbeater.cc:475] Master 127.12.131.62:52835 requested a full tablet report, sending... I0627 19:31:25.455284 20037 ts_manager.cc:98] Registered new tserver with Master: 0090108d1d7d43868dae9f524043684e (127.12.131.3:57976) I0627 19:31:25.459848 12812 external_mini_cluster.cc:490] 3 TS(s) registered with all masters I0627 19:31:25.486402 20531 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.478344 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:44306 0627 19:31:25.478653 (+ 309us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:25.478677 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:25.481927 (+ 3250us) server_negotiation.cc:373] Connection header received 0627 19:31:25.482056 (+ 129us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:25.482082 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:25.482197 (+ 115us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:25.482324 (+ 127us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:25.483400 (+ 1076us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:25.483427 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:25.483473 (+ 46us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:25.483526 (+ 53us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:25.483589 (+ 63us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:25.483644 (+ 55us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:25.483751 (+ 107us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:25.485145 (+ 1394us) server_negotiation.cc:264] Negotiation successful 0627 19:31:25.485241 (+ 96us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":122} I0627 19:31:25.492027 20037 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.12.131.62:44306: name: "test-table" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "val" type: INT32 is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0627 19:31:25.509117 20037 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0627 19:31:25.510686 20398 heartbeater.cc:467] Master 127.12.131.62:52835 was elected leader, sending a full tablet report... I0627 19:31:25.547240 20542 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.530011 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:37617 0627 19:31:25.532207 (+ 2196us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:25.532237 (+ 30us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:25.533795 (+ 1558us) server_negotiation.cc:373] Connection header received 0627 19:31:25.534173 (+ 378us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:25.534202 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:25.534368 (+ 166us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:25.534576 (+ 208us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:25.544643 (+ 10067us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:25.544687 (+ 44us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:25.544715 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:25.544767 (+ 52us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:25.544867 (+ 100us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:25.544912 (+ 45us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:25.545036 (+ 124us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:25.545754 (+ 718us) server_negotiation.cc:264] Negotiation successful 0627 19:31:25.545875 (+ 121us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1923,"thread_start_us":2158,"threads_started":1} I0627 19:31:25.548995 20543 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.529871 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:43707 0627 19:31:25.543950 (+ 14079us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:25.543983 (+ 33us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:25.544033 (+ 50us) server_negotiation.cc:373] Connection header received 0627 19:31:25.544163 (+ 130us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:25.544191 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:25.544421 (+ 230us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:25.545141 (+ 720us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:25.546756 (+ 1615us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:25.546790 (+ 34us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:25.546816 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:25.546866 (+ 50us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:25.546961 (+ 95us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:25.547021 (+ 60us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:25.547135 (+ 114us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:25.548028 (+ 893us) server_negotiation.cc:264] Negotiation successful 0627 19:31:25.548109 (+ 81us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5712,"thread_start_us":3248,"threads_started":1} I0627 19:31:25.550956 20544 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.529064 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.1:33551 0627 19:31:25.533569 (+ 4505us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:25.533662 (+ 93us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:25.533935 (+ 273us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:25.543270 (+ 9335us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:25.543299 (+ 29us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:25.543416 (+ 117us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:25.543524 (+ 108us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0627 19:31:25.543547 (+ 23us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:25.543596 (+ 49us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0627 19:31:25.543719 (+ 123us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:25.545042 (+ 1323us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:25.545076 (+ 34us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:25.545104 (+ 28us) client_negotiation.cc:695] Sending connection context 0627 19:31:25.547839 (+ 2735us) client_negotiation.cc:225] Negotiation successful 0627 19:31:25.549505 (+ 1666us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":4136,"thread_start_us":5757,"threads_started":1} I0627 19:31:25.550998 20541 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.530667 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.3:57976 0627 19:31:25.535467 (+ 4800us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:25.535611 (+ 144us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:25.535719 (+ 108us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:25.546158 (+ 10439us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:25.546185 (+ 27us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:25.546248 (+ 63us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:25.546332 (+ 84us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0627 19:31:25.546352 (+ 20us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:25.546411 (+ 59us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0627 19:31:25.546507 (+ 96us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:25.547241 (+ 734us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:25.547271 (+ 30us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:25.547295 (+ 24us) client_negotiation.cc:695] Sending connection context 0627 19:31:25.547559 (+ 264us) client_negotiation.cc:225] Negotiation successful 0627 19:31:25.547704 (+ 145us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":4055,"thread_start_us":1407,"threads_started":1} I0627 19:31:25.553655 20462 tablet_service.cc:759] Processing CreateTablet for tablet df06cce11c6b4e8fb458f502521b5572 (table=test-table [id=d1121bcc4779447c9768cccab464d4d6]), partition=RANGE (key) PARTITION UNBOUNDED I0627 19:31:25.553831 20207 tablet_service.cc:759] Processing CreateTablet for tablet df06cce11c6b4e8fb458f502521b5572 (table=test-table [id=d1121bcc4779447c9768cccab464d4d6]), partition=RANGE (key) PARTITION UNBOUNDED I0627 19:31:25.554615 20462 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet df06cce11c6b4e8fb458f502521b5572. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:25.554723 20207 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet df06cce11c6b4e8fb458f502521b5572. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:25.559798 20546 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.536008 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:36415 0627 19:31:25.542785 (+ 6777us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:25.542820 (+ 35us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:25.542873 (+ 53us) server_negotiation.cc:373] Connection header received 0627 19:31:25.543038 (+ 165us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:25.543075 (+ 37us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:25.543238 (+ 163us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:25.543411 (+ 173us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:25.556385 (+ 12974us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:25.556449 (+ 64us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:25.556481 (+ 32us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:25.556539 (+ 58us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:25.556654 (+ 115us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:25.556724 (+ 70us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:25.556855 (+ 131us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:25.558355 (+ 1500us) server_negotiation.cc:264] Negotiation successful 0627 19:31:25.558507 (+ 152us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":6510,"thread_start_us":6838,"threads_started":1} I0627 19:31:25.561329 20545 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.535921 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.2:60127 0627 19:31:25.542248 (+ 6327us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:25.542391 (+ 143us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:25.542527 (+ 136us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:25.555500 (+ 12973us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:25.555530 (+ 30us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:25.555606 (+ 76us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:25.555704 (+ 98us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0627 19:31:25.555722 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:25.555792 (+ 70us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0627 19:31:25.555895 (+ 103us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:25.556994 (+ 1099us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:25.557021 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:25.557046 (+ 25us) client_negotiation.cc:695] Sending connection context 0627 19:31:25.557144 (+ 98us) client_negotiation.cc:225] Negotiation successful 0627 19:31:25.557281 (+ 137us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":6069,"thread_start_us":5670,"threads_started":1} I0627 19:31:25.564250 20335 tablet_service.cc:759] Processing CreateTablet for tablet df06cce11c6b4e8fb458f502521b5572 (table=test-table [id=d1121bcc4779447c9768cccab464d4d6]), partition=RANGE (key) PARTITION UNBOUNDED I0627 19:31:25.565280 20335 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet df06cce11c6b4e8fb458f502521b5572. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:25.583891 20462 ts_tablet_manager.cc:1173] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: Registered tablet (data state: TABLET_DATA_READY) I0627 19:31:25.590098 20547 ts_tablet_manager.cc:1028] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: Bootstrapping tablet I0627 19:31:25.594297 20547 tablet_bootstrap.cc:427] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: Bootstrap starting. I0627 19:31:25.598420 20547 tablet_bootstrap.cc:578] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: No blocks or log segments found. Creating new log. I0627 19:31:25.600473 20547 log.cc:525] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: Log is configured to *not* fsync() on all Append() calls I0627 19:31:25.610646 20547 tablet_bootstrap.cc:427] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: No bootstrap required, opened a new log I0627 19:31:25.611186 20547 ts_tablet_manager.cc:1045] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s I0627 19:31:25.632786 20207 ts_tablet_manager.cc:1173] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: Registered tablet (data state: TABLET_DATA_READY) I0627 19:31:25.633514 20547 raft_consensus.cc:325] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7bb63625d4482fb8e1a864a845255d" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 33551 } } peers { permanent_uuid: "0090108d1d7d43868dae9f524043684e" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57976 } } peers { permanent_uuid: "87ef418b73d04687969fbd353b1085cd" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60127 } } I0627 19:31:25.634551 20547 raft_consensus.cc:351] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:25.634804 20547 raft_consensus.cc:628] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0090108d1d7d43868dae9f524043684e, State: Initialized, Role: FOLLOWER I0627 19:31:25.635653 20547 consensus_queue.cc:227] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7bb63625d4482fb8e1a864a845255d" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 33551 } } peers { permanent_uuid: "0090108d1d7d43868dae9f524043684e" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57976 } } peers { permanent_uuid: "87ef418b73d04687969fbd353b1085cd" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60127 } } I0627 19:31:25.642587 20335 ts_tablet_manager.cc:1173] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Registered tablet (data state: TABLET_DATA_READY) I0627 19:31:25.645577 20526 heartbeater.cc:467] Master 127.12.131.62:52835 was elected leader, sending a full tablet report... I0627 19:31:25.646548 20547 ts_tablet_manager.cc:1073] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: Time spent starting tablet: real 0.035s user 0.019s sys 0.009s I0627 19:31:25.652042 20550 ts_tablet_manager.cc:1028] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: Bootstrapping tablet I0627 19:31:25.652551 20551 ts_tablet_manager.cc:1028] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Bootstrapping tablet I0627 19:31:25.652683 20550 tablet_bootstrap.cc:427] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: Bootstrap starting. I0627 19:31:25.653115 20551 tablet_bootstrap.cc:427] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Bootstrap starting. I0627 19:31:25.657349 20551 tablet_bootstrap.cc:578] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: No blocks or log segments found. Creating new log. I0627 19:31:25.657821 20550 tablet_bootstrap.cc:578] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: No blocks or log segments found. Creating new log. I0627 19:31:25.659019 20551 log.cc:525] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Log is configured to *not* fsync() on all Append() calls I0627 19:31:25.659710 20550 log.cc:525] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: Log is configured to *not* fsync() on all Append() calls I0627 19:31:25.663980 20550 tablet_bootstrap.cc:427] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: No bootstrap required, opened a new log I0627 19:31:25.664496 20550 ts_tablet_manager.cc:1045] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s I0627 19:31:25.671012 20551 tablet_bootstrap.cc:427] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: No bootstrap required, opened a new log I0627 19:31:25.671501 20551 ts_tablet_manager.cc:1045] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.004s I0627 19:31:25.697896 20550 raft_consensus.cc:325] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7bb63625d4482fb8e1a864a845255d" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 33551 } } peers { permanent_uuid: "0090108d1d7d43868dae9f524043684e" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57976 } } peers { permanent_uuid: "87ef418b73d04687969fbd353b1085cd" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60127 } } I0627 19:31:25.709791 20550 raft_consensus.cc:351] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:25.710129 20550 raft_consensus.cc:628] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7bb63625d4482fb8e1a864a845255d, State: Initialized, Role: FOLLOWER I0627 19:31:25.710985 20551 raft_consensus.cc:325] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7bb63625d4482fb8e1a864a845255d" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 33551 } } peers { permanent_uuid: "0090108d1d7d43868dae9f524043684e" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57976 } } peers { permanent_uuid: "87ef418b73d04687969fbd353b1085cd" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60127 } } I0627 19:31:25.711923 20550 consensus_queue.cc:227] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7bb63625d4482fb8e1a864a845255d" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 33551 } } peers { permanent_uuid: "0090108d1d7d43868dae9f524043684e" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57976 } } peers { permanent_uuid: "87ef418b73d04687969fbd353b1085cd" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60127 } } I0627 19:31:25.713732 20551 raft_consensus.cc:351] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:25.713981 20551 raft_consensus.cc:628] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87ef418b73d04687969fbd353b1085cd, State: Initialized, Role: FOLLOWER I0627 19:31:25.714850 20551 consensus_queue.cc:227] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7bb63625d4482fb8e1a864a845255d" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 33551 } } peers { permanent_uuid: "0090108d1d7d43868dae9f524043684e" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57976 } } peers { permanent_uuid: "87ef418b73d04687969fbd353b1085cd" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60127 } } I0627 19:31:25.724519 20550 ts_tablet_manager.cc:1073] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: Time spent starting tablet: real 0.060s user 0.020s sys 0.004s I0627 19:31:25.726346 20555 raft_consensus.cc:457] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0627 19:31:25.726753 20555 raft_consensus.cc:2730] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Snoozing failure detection for 2.109s (starting election) I0627 19:31:25.727164 20555 raft_consensus.cc:479] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7bb63625d4482fb8e1a864a845255d" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 33551 } } peers { permanent_uuid: "0090108d1d7d43868dae9f524043684e" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57976 } } peers { permanent_uuid: "87ef418b73d04687969fbd353b1085cd" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60127 } } I0627 19:31:25.742722 20555 leader_election.cc:243] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551) I0627 19:31:25.743541 20555 leader_election.cc:243] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0090108d1d7d43868dae9f524043684e (127.12.131.3:57976) W0627 19:31:25.670411 20527 tablet.cc:1678] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:25.727084 20551 ts_tablet_manager.cc:1073] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Time spent starting tablet: real 0.055s user 0.020s sys 0.004s I0627 19:31:25.918931 20543 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.744605 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:47630 0627 19:31:25.744948 (+ 343us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:25.744969 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:25.915961 (+170992us) server_negotiation.cc:373] Connection header received 0627 19:31:25.916181 (+ 220us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:25.916208 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:25.916317 (+ 109us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:25.916473 (+ 156us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:25.917083 (+ 610us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:25.917109 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:25.917132 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:25.917186 (+ 54us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:25.917250 (+ 64us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:25.917303 (+ 53us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:25.917437 (+ 134us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:25.917696 (+ 259us) server_negotiation.cc:264] Negotiation successful 0627 19:31:25.917786 (+ 90us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":133} I0627 19:31:25.919085 20557 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.744368 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.3:57976 0627 19:31:25.915786 (+171418us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:25.915859 (+ 73us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:25.915987 (+ 128us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:25.916515 (+ 528us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:25.916540 (+ 25us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:25.916598 (+ 58us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:25.916722 (+ 124us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0627 19:31:25.916739 (+ 17us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:25.916802 (+ 63us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0627 19:31:25.916905 (+ 103us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:25.917441 (+ 536us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:25.917466 (+ 25us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:25.917490 (+ 24us) client_negotiation.cc:695] Sending connection context 0627 19:31:25.917594 (+ 104us) client_negotiation.cc:225] Negotiation successful 0627 19:31:25.917727 (+ 133us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":171079,"thread_start_us":170732,"threads_started":1} I0627 19:31:25.920233 20482 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "df06cce11c6b4e8fb458f502521b5572" candidate_uuid: "87ef418b73d04687969fbd353b1085cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0090108d1d7d43868dae9f524043684e" is_pre_election: true I0627 19:31:25.920879 20482 raft_consensus.cc:2730] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: Snoozing failure detection for 1.783s (vote granted) I0627 19:31:25.921151 20482 raft_consensus.cc:2243] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ef418b73d04687969fbd353b1085cd in term 0. I0627 19:31:25.923223 20288 leader_election.cc:379] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [CANDIDATE]: Term 1 pre-election: Vote granted by peer 0090108d1d7d43868dae9f524043684e (127.12.131.3:57976) I0627 19:31:25.923647 20288 leader_election.cc:270] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0627 19:31:25.929102 20556 raft_consensus.cc:2730] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Snoozing failure detection for 1.847s (election complete) I0627 19:31:25.929599 20556 raft_consensus.cc:2537] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 0 FOLLOWER]: Leader pre-election won for term 1 I0627 19:31:25.929842 20556 raft_consensus.cc:457] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0627 19:31:25.930052 20556 raft_consensus.cc:2730] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Snoozing failure detection for 1.660s (starting election) I0627 19:31:25.930268 20556 raft_consensus.cc:2782] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:25.930732 20542 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.915556 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:50646 0627 19:31:25.915899 (+ 343us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:25.915922 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:25.919674 (+ 3752us) server_negotiation.cc:373] Connection header received 0627 19:31:25.922991 (+ 3317us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:25.923017 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:25.923122 (+ 105us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:25.923252 (+ 130us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:25.923928 (+ 676us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:25.923956 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:25.923978 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:25.924026 (+ 48us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:25.924073 (+ 47us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:25.924110 (+ 37us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:25.924205 (+ 95us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:25.929483 (+ 5278us) server_negotiation.cc:264] Negotiation successful 0627 19:31:25.929578 (+ 95us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":137} I0627 19:31:25.935739 20556 raft_consensus.cc:479] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7bb63625d4482fb8e1a864a845255d" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 33551 } } peers { permanent_uuid: "0090108d1d7d43868dae9f524043684e" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57976 } } peers { permanent_uuid: "87ef418b73d04687969fbd353b1085cd" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60127 } } I0627 19:31:25.940073 20557 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.915722 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.1:33551 0627 19:31:25.919568 (+ 3846us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:25.919602 (+ 34us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:25.921699 (+ 2097us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:25.923320 (+ 1621us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:25.923346 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:25.923463 (+ 117us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:25.923527 (+ 64us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0627 19:31:25.923555 (+ 28us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:25.923584 (+ 29us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0627 19:31:25.923750 (+ 166us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:25.924246 (+ 496us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:25.924274 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:25.924399 (+ 125us) client_negotiation.cc:695] Sending connection context 0627 19:31:25.930756 (+ 6357us) client_negotiation.cc:225] Negotiation successful 0627 19:31:25.931166 (+ 410us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":3656,"thread_start_us":5396,"threads_started":1} I0627 19:31:25.937072 20556 leader_election.cc:243] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [CANDIDATE]: Term 1 election: Requesting vote from peer 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551) I0627 19:31:25.940932 20556 leader_election.cc:243] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [CANDIDATE]: Term 1 election: Requesting vote from peer 0090108d1d7d43868dae9f524043684e (127.12.131.3:57976) I0627 19:31:25.942591 20482 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "df06cce11c6b4e8fb458f502521b5572" candidate_uuid: "87ef418b73d04687969fbd353b1085cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0090108d1d7d43868dae9f524043684e" I0627 19:31:25.942608 20227 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "df06cce11c6b4e8fb458f502521b5572" candidate_uuid: "87ef418b73d04687969fbd353b1085cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e7bb63625d4482fb8e1a864a845255d" is_pre_election: true I0627 19:31:25.942932 20482 raft_consensus.cc:2782] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:25.943154 20482 raft_consensus.cc:2730] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: Snoozing failure detection for 1.597s (vote granted) I0627 19:31:25.946529 20482 raft_consensus.cc:2243] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ef418b73d04687969fbd353b1085cd in term 1. I0627 19:31:25.947269 20288 leader_election.cc:379] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [CANDIDATE]: Term 1 election: Vote granted by peer 0090108d1d7d43868dae9f524043684e (127.12.131.3:57976) I0627 19:31:25.947552 20288 leader_election.cc:270] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:25.948029 20556 raft_consensus.cc:2730] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Snoozing failure detection for 1.881s (election complete) I0627 19:31:25.948307 20556 raft_consensus.cc:2537] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:25.948845 20556 raft_consensus.cc:593] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 1 LEADER]: Becoming Leader. State: Replica: 87ef418b73d04687969fbd353b1085cd, State: Running, Role: LEADER I0627 19:31:25.951678 20556 consensus_queue.cc:204] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7bb63625d4482fb8e1a864a845255d" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 33551 } } peers { permanent_uuid: "0090108d1d7d43868dae9f524043684e" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57976 } } peers { permanent_uuid: "87ef418b73d04687969fbd353b1085cd" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60127 } } I0627 19:31:25.964155 20037 catalog_manager.cc:3798] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd reported cstate change: term changed from 0 to 1, leader changed from to 87ef418b73d04687969fbd353b1085cd (127.12.131.2). New cstate: current_term: 1 leader_uuid: "87ef418b73d04687969fbd353b1085cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7bb63625d4482fb8e1a864a845255d" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 33551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0090108d1d7d43868dae9f524043684e" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57976 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87ef418b73d04687969fbd353b1085cd" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60127 } health_report { overall_health: HEALTHY } } } W0627 19:31:25.976512 20399 tablet.cc:1678] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:25.833148 20554 raft_consensus.cc:457] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0627 19:31:25.983582 20554 raft_consensus.cc:2730] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: Snoozing failure detection for 2.193s (starting election) I0627 19:31:25.983925 20554 raft_consensus.cc:479] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e7bb63625d4482fb8e1a864a845255d" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 33551 } } peers { permanent_uuid: "0090108d1d7d43868dae9f524043684e" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57976 } } peers { permanent_uuid: "87ef418b73d04687969fbd353b1085cd" member_type: VOTER last_known_addr { host: "127.12.131.2" port: 60127 } } I0627 19:31:25.985473 20227 raft_consensus.cc:2730] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: Snoozing failure detection for 2.206s (vote granted) I0627 19:31:25.985699 20554 leader_election.cc:243] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0090108d1d7d43868dae9f524043684e (127.12.131.3:57976) I0627 19:31:25.985723 20227 raft_consensus.cc:2243] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87ef418b73d04687969fbd353b1085cd in term 0. I0627 19:31:25.986620 20554 leader_election.cc:243] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 87ef418b73d04687969fbd353b1085cd (127.12.131.2:60127) W0627 19:31:25.839171 20273 tablet.cc:1678] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:26.033452 20564 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:26.015123 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:36416 0627 19:31:26.029170 (+ 14047us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:26.029196 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:26.029236 (+ 40us) server_negotiation.cc:373] Connection header received 0627 19:31:26.029406 (+ 170us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:26.029434 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:26.029747 (+ 313us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:26.030516 (+ 769us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:26.030800 (+ 284us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:26.030827 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:26.030850 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:26.030896 (+ 46us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:26.030956 (+ 60us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:26.031010 (+ 54us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:26.031765 (+ 755us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:26.031919 (+ 154us) server_negotiation.cc:264] Negotiation successful 0627 19:31:26.032008 (+ 89us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":13827,"thread_start_us":14402,"threads_started":1} I0627 19:31:26.056574 20482 raft_consensus.cc:1081] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e [term 1 FOLLOWER]: Refusing update from remote peer 87ef418b73d04687969fbd353b1085cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0627 19:31:26.057955 20556 consensus_queue.cc:962] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0090108d1d7d43868dae9f524043684e" member_type: VOTER last_known_addr { host: "127.12.131.3" port: 57976 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0627 19:31:25.943629 20226 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "df06cce11c6b4e8fb458f502521b5572" candidate_uuid: "87ef418b73d04687969fbd353b1085cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e7bb63625d4482fb8e1a864a845255d" I0627 19:31:26.059789 20226 raft_consensus.cc:2782] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:26.060086 20226 raft_consensus.cc:2730] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: Snoozing failure detection for 2.168s (vote granted) I0627 19:31:26.066138 20226 raft_consensus.cc:2243] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87ef418b73d04687969fbd353b1085cd in term 1. I0627 19:31:26.073871 20561 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.987499 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.2:60127 0627 19:31:26.070326 (+ 82827us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:26.070396 (+ 70us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:26.070518 (+ 122us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:26.071290 (+ 772us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:26.071313 (+ 23us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:26.071395 (+ 82us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:26.071522 (+ 127us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0627 19:31:26.071540 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:26.071575 (+ 35us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0627 19:31:26.071670 (+ 95us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:26.072283 (+ 613us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:26.072309 (+ 26us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:26.072332 (+ 23us) client_negotiation.cc:695] Sending connection context 0627 19:31:26.072456 (+ 124us) client_negotiation.cc:225] Negotiation successful 0627 19:31:26.072589 (+ 133us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":82560,"threads_started":1} I0627 19:31:26.075883 20546 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.988142 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:43381 0627 19:31:25.988491 (+ 349us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:25.988515 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:26.070693 (+ 82178us) server_negotiation.cc:373] Connection header received 0627 19:31:26.070818 (+ 125us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:26.070851 (+ 33us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:26.070964 (+ 113us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:26.071086 (+ 122us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:26.071882 (+ 796us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:26.071908 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:26.071929 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:26.071979 (+ 50us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:26.072037 (+ 58us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:26.072087 (+ 50us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:26.072183 (+ 96us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:26.074677 (+ 2494us) server_negotiation.cc:264] Negotiation successful 0627 19:31:26.074768 (+ 91us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":104} I0627 19:31:26.078305 20355 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "df06cce11c6b4e8fb458f502521b5572" candidate_uuid: "3e7bb63625d4482fb8e1a864a845255d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87ef418b73d04687969fbd353b1085cd" is_pre_election: true I0627 19:31:26.079044 20355 raft_consensus.cc:2204] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 3e7bb63625d4482fb8e1a864a845255d for term 1 because replica is either leader or believes a valid leader to be alive. I0627 19:31:26.079190 20561 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.989211 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.3:57976 0627 19:31:26.074271 (+ 85060us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:26.074304 (+ 33us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:26.074431 (+ 127us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:26.076965 (+ 2534us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:26.076990 (+ 25us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:26.077046 (+ 56us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:26.077094 (+ 48us) client_negotiation.cc:552] Initiating SASL PLAIN handshake 0627 19:31:26.077112 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:26.077137 (+ 25us) client_negotiation.cc:740] callback for SASL_CB_AUTHNAME 0627 19:31:26.077232 (+ 95us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:26.077766 (+ 534us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:26.077793 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:26.077816 (+ 23us) client_negotiation.cc:695] Sending connection context 0627 19:31:26.077919 (+ 103us) client_negotiation.cc:225] Negotiation successful 0627 19:31:26.078032 (+ 113us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":84883,"thread_start_us":80575,"threads_started":1} I0627 19:31:26.079246 20543 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:25.988966 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:38873 0627 19:31:25.989295 (+ 329us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:25.989318 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:26.074385 (+ 85067us) server_negotiation.cc:373] Connection header received 0627 19:31:26.074600 (+ 215us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:26.074625 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:26.074725 (+ 100us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:26.074845 (+ 120us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:26.077434 (+ 2589us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:26.077460 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:26.077481 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:26.077526 (+ 45us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:26.077585 (+ 59us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:26.077634 (+ 49us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:26.077744 (+ 110us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:26.078017 (+ 273us) server_negotiation.cc:264] Negotiation successful 0627 19:31:26.078104 (+ 87us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":136} I0627 19:31:26.080709 20285 leader_election.cc:379] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [CANDIDATE]: Term 1 pre-election: Vote granted by peer 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551) I0627 19:31:26.080557 20481 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "df06cce11c6b4e8fb458f502521b5572" candidate_uuid: "3e7bb63625d4482fb8e1a864a845255d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0090108d1d7d43868dae9f524043684e" is_pre_election: true I0627 19:31:26.081188 20226 raft_consensus.cc:1081] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [term 1 FOLLOWER]: Refusing update from remote peer 87ef418b73d04687969fbd353b1085cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0627 19:31:26.082020 20285 leader_election.cc:379] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [CANDIDATE]: Term 1 election: Vote granted by peer 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551) I0627 19:31:26.082931 20157 leader_election.cc:393] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [CANDIDATE]: Term 1 pre-election: Vote denied by peer 87ef418b73d04687969fbd353b1085cd (127.12.131.2:60127). Message: Invalid argument: T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 3e7bb63625d4482fb8e1a864a845255d for term 1 because replica is either leader or believes a valid leader to be alive. I0627 19:31:26.083974 20481 raft_consensus.cc:2218] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e7bb63625d4482fb8e1a864a845255d for term 1 because replica is already servicing an update from a current leader or another vote. I0627 19:31:26.083737 20556 consensus_queue.cc:962] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e7bb63625d4482fb8e1a864a845255d" member_type: VOTER last_known_addr { host: "127.12.131.1" port: 33551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0627 19:31:26.094558 20160 leader_election.cc:393] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [CANDIDATE]: Term 1 pre-election: Vote denied by peer 0090108d1d7d43868dae9f524043684e (127.12.131.3:57976). Message: Service unavailable: T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3e7bb63625d4482fb8e1a864a845255d for term 1 because replica is already servicing an update from a current leader or another vote. I0627 19:31:26.094862 20160 leader_election.cc:270] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0627 19:31:26.120707 20554 raft_consensus.cc:2730] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d: Snoozing failure detection for 1.742s (election complete) I0627 19:31:26.120949 20554 raft_consensus.cc:2485] T df06cce11c6b4e8fb458f502521b5572 P 3e7bb63625d4482fb8e1a864a845255d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0627 19:31:26.128291 20037 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.12.131.62:44306: table { table_name: "test-table" } I0627 19:31:26.139127 20035 catalog_manager.cc:3675] Got report from deleted tablet df06cce11c6b4e8fb458f502521b5572 (table test-table [id=d1121bcc4779447c9768cccab464d4d6]) (Table deleted at 2018-06-27 19:31:26 PDT): Sending delete request for this tablet I0627 19:31:26.139788 20037 catalog_manager.cc:4037] T 00000000000000000000000000000000 P 6c0d2724a2fb42f7bb0f41b0251080bf: Sending DeleteTablet for 3 replicas of tablet df06cce11c6b4e8fb458f502521b5572 I0627 19:31:26.140051 20035 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df06cce11c6b4e8fb458f502521b5572 on 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551) (Table deleted at 2018-06-27 19:31:26 PDT) I0627 19:31:26.140154 20037 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df06cce11c6b4e8fb458f502521b5572 on 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551) (Table deleted at 2018-06-27 19:31:26 PDT) I0627 19:31:26.140699 20037 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df06cce11c6b4e8fb458f502521b5572 on 0090108d1d7d43868dae9f524043684e (127.12.131.3:57976) (Table deleted at 2018-06-27 19:31:26 PDT) I0627 19:31:26.141115 20037 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df06cce11c6b4e8fb458f502521b5572 on 87ef418b73d04687969fbd353b1085cd (127.12.131.2:60127) (Table deleted at 2018-06-27 19:31:26 PDT) I0627 19:31:26.142273 20335 tablet_service.cc:800] Processing DeleteTablet for tablet df06cce11c6b4e8fb458f502521b5572 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-27 19:31:26 PDT) from {username='jenkins'} at 127.12.131.2:36415 I0627 19:31:26.142235 20206 tablet_service.cc:800] Processing DeleteTablet for tablet df06cce11c6b4e8fb458f502521b5572 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-27 19:31:26 PDT) from {username='jenkins'} at 127.12.131.1:37617 I0627 19:31:26.142496 20207 tablet_service.cc:800] Processing DeleteTablet for tablet df06cce11c6b4e8fb458f502521b5572 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-27 19:31:26 PDT) from {username='jenkins'} at 127.12.131.1:37617 I0627 19:31:26.143005 20462 tablet_service.cc:800] Processing DeleteTablet for tablet df06cce11c6b4e8fb458f502521b5572 with delete_type TABLET_DATA_DELETED (Table deleted at 2018-06-27 19:31:26 PDT) from {username='jenkins'} at 127.12.131.3:43707 I0627 19:31:26.146260 20575 tablet_replica.cc:261] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: stopping tablet replica I0627 19:31:26.146944 20575 maintenance_manager.cc:237] P 0090108d1d7d43868dae9f524043684e: Unregistered op CompactRowSetsOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.147208 20575 maintenance_manager.cc:237] P 0090108d1d7d43868dae9f524043684e: Unregistered op MinorDeltaCompactionOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.146283 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 20111 I0627 19:31:26.147459 20575 maintenance_manager.cc:237] P 0090108d1d7d43868dae9f524043684e: Unregistered op MajorDeltaCompactionOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.147984 20575 maintenance_manager.cc:237] P 0090108d1d7d43868dae9f524043684e: Unregistered op UndoDeltaBlockGCOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.148244 20575 maintenance_manager.cc:237] P 0090108d1d7d43868dae9f524043684e: Unregistered op FlushMRSOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.148838 20575 maintenance_manager.cc:237] P 0090108d1d7d43868dae9f524043684e: Unregistered op FlushDeltaMemStoresOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.149050 20575 maintenance_manager.cc:237] P 0090108d1d7d43868dae9f524043684e: Unregistered op LogGCOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.149291 20575 raft_consensus.cc:2040] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e [term 1 FOLLOWER]: Raft consensus shutting down. I0627 19:31:26.149673 20575 raft_consensus.cc:2067] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e [term 1 FOLLOWER]: Raft consensus is shut down! I0627 19:31:26.153302 20575 ts_tablet_manager.cc:1369] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: Deleting tablet data with delete state TABLET_DATA_DELETED I0627 19:31:26.164487 20575 ts_tablet_manager.cc:1382] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0627 19:31:26.165351 20576 tablet_replica.cc:261] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: stopping tablet replica I0627 19:31:26.165735 20576 maintenance_manager.cc:237] P 87ef418b73d04687969fbd353b1085cd: Unregistered op CompactRowSetsOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.165938 20576 maintenance_manager.cc:237] P 87ef418b73d04687969fbd353b1085cd: Unregistered op MinorDeltaCompactionOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.166124 20576 maintenance_manager.cc:237] P 87ef418b73d04687969fbd353b1085cd: Unregistered op MajorDeltaCompactionOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.166307 20576 maintenance_manager.cc:237] P 87ef418b73d04687969fbd353b1085cd: Unregistered op UndoDeltaBlockGCOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.166550 20576 maintenance_manager.cc:237] P 87ef418b73d04687969fbd353b1085cd: Unregistered op FlushMRSOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.166728 20576 maintenance_manager.cc:237] P 87ef418b73d04687969fbd353b1085cd: Unregistered op FlushDeltaMemStoresOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.166883 20576 maintenance_manager.cc:237] P 87ef418b73d04687969fbd353b1085cd: Unregistered op LogGCOp(df06cce11c6b4e8fb458f502521b5572) I0627 19:31:26.167083 20576 raft_consensus.cc:2040] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 1 LEADER]: Raft consensus shutting down. I0627 19:31:26.167286 20576 consensus_peers.cc:455] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd -> Peer 0090108d1d7d43868dae9f524043684e (127.12.131.3:57976): Closing peer: 0090108d1d7d43868dae9f524043684e I0627 19:31:26.169713 20576 consensus_peers.cc:455] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd -> Peer 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551): Closing peer: 3e7bb63625d4482fb8e1a864a845255d I0627 19:31:26.170401 20576 raft_consensus.cc:2067] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd [term 1 FOLLOWER]: Raft consensus is shut down! I0627 19:31:26.172963 20575 log.cc:986] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-2/wal/wals/df06cce11c6b4e8fb458f502521b5572 I0627 19:31:26.174440 20576 ts_tablet_manager.cc:1369] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Deleting tablet data with delete state TABLET_DATA_DELETED I0627 19:31:26.176589 20575 ts_tablet_manager.cc:1403] T df06cce11c6b4e8fb458f502521b5572 P 0090108d1d7d43868dae9f524043684e: Deleting consensus metadata I0627 19:31:26.179986 20004 catalog_manager.cc:3103] TS 0090108d1d7d43868dae9f524043684e (127.12.131.3:57976): tablet df06cce11c6b4e8fb458f502521b5572 (table test-table [id=d1121bcc4779447c9768cccab464d4d6]) successfully deleted I0627 19:31:26.200404 20576 ts_tablet_manager.cc:1382] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2 I0627 19:31:26.200779 20576 log.cc:986] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1530153042096250-12812/minicluster-data/ts-1/wal/wals/df06cce11c6b4e8fb458f502521b5572 I0627 19:31:26.201638 20576 ts_tablet_manager.cc:1403] T df06cce11c6b4e8fb458f502521b5572 P 87ef418b73d04687969fbd353b1085cd: Deleting consensus metadata I0627 19:31:26.205142 19959 catalog_manager.cc:3103] TS 87ef418b73d04687969fbd353b1085cd (127.12.131.2:60127): tablet df06cce11c6b4e8fb458f502521b5572 (table test-table [id=d1121bcc4779447c9768cccab464d4d6]) successfully deleted W0627 19:31:26.207286 19959 catalog_manager.cc:2867] TS 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551): Delete Tablet RPC failed for tablet df06cce11c6b4e8fb458f502521b5572: Network error: recv got EOF from 127.12.131.1:33551 (error 108) I0627 19:31:26.207569 19959 catalog_manager.cc:2903] Scheduling retry of df06cce11c6b4e8fb458f502521b5572 Delete Tablet RPC for TS=3e7bb63625d4482fb8e1a864a845255d with a delay of 32 ms (attempt = 1) W0627 19:31:26.207829 19959 catalog_manager.cc:2867] TS 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551): Delete Tablet RPC failed for tablet df06cce11c6b4e8fb458f502521b5572: Network error: recv got EOF from 127.12.131.1:33551 (error 108) I0627 19:31:26.207952 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 20272 I0627 19:31:26.208042 19959 catalog_manager.cc:2903] Scheduling retry of df06cce11c6b4e8fb458f502521b5572 Delete Tablet RPC for TS=3e7bb63625d4482fb8e1a864a845255d with a delay of 63 ms (attempt = 1) I0627 19:31:26.242161 19959 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df06cce11c6b4e8fb458f502521b5572 on 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551) (Table deleted at 2018-06-27 19:31:26 PDT) I0627 19:31:26.245717 20577 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:26.242826 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.1:33551 0627 19:31:26.244998 (+ 2172us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:26.245165 (+ 167us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.1:33551: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":1967,"thread_start_us":2347,"threads_started":1} W0627 19:31:26.246031 19959 catalog_manager.cc:2867] TS 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551): Delete Tablet RPC failed for tablet df06cce11c6b4e8fb458f502521b5572: Network error: Client connection negotiation failed: client connection to 127.12.131.1:33551: connect: Connection refused (error 111) I0627 19:31:26.246299 19959 catalog_manager.cc:2903] Scheduling retry of df06cce11c6b4e8fb458f502521b5572 Delete Tablet RPC for TS=3e7bb63625d4482fb8e1a864a845255d with a delay of 80 ms (attempt = 2) I0627 19:31:26.247174 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 20402 I0627 19:31:26.272413 19959 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet df06cce11c6b4e8fb458f502521b5572 on 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551) (Table deleted at 2018-06-27 19:31:26 PDT) I0627 19:31:26.274294 20577 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:26.273033 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.1:33551 0627 19:31:26.273688 (+ 655us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:26.273848 (+ 160us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.1:33551: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":417} W0627 19:31:26.274600 19959 catalog_manager.cc:2867] TS 3e7bb63625d4482fb8e1a864a845255d (127.12.131.1:33551): Delete Tablet RPC failed for tablet df06cce11c6b4e8fb458f502521b5572: Network error: Client connection negotiation failed: client connection to 127.12.131.1:33551: connect: Connection refused (error 111) I0627 19:31:26.274837 19959 catalog_manager.cc:2903] Scheduling retry of df06cce11c6b4e8fb458f502521b5572 Delete Tablet RPC for TS=3e7bb63625d4482fb8e1a864a845255d with a delay of 79 ms (attempt = 2) I0627 19:31:26.281800 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 19916 [ OK ] SecurityITest.TestDisableAuthenticationEncryption (6831 ms) [ RUN ] SecurityITest.TestWorldReadableKeytab [ OK ] SecurityITest.TestWorldReadableKeytab (272 ms) [ RUN ] SecurityITest.TestWorldReadablePrivateKey [ OK ] SecurityITest.TestWorldReadablePrivateKey (273 ms) [----------] 9 tests from SecurityITest (44665 ms total) [----------] 12 tests from AuthTokenIssuingTest [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:27 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20583](info): setting up network... Jun 27 19:31:27 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20583](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:27 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20583](info): skipping unrecognized local address family 17 Jun 27 19:31:27 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20583](info): skipping unrecognized local address family 17 Jun 27 19:31:27 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20583](info): set up 1 sockets Jun 27 19:31:27 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20583](info): commencing operation krb5kdc: starting... W0627 19:31:27.107764 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.203s user 0.003s sys 0.013s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:27 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.0.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. I0627 19:31:27.302954 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:46652 --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=required --rpc-encryption=required --rpc_encrypt_loopback_connections=true --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:27.557145 20592 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:27.557646 20592 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.0.0.1 W0627 19:31:27.558073 20592 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:27.576957 20592 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:27.577235 20592 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:27.577553 20592 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:27.577741 20592 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:27.577958 20592 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:27.578188 20592 flags.cc:406] Enabled experimental flag: --principal=kudu/127.0.0.1 I0627 19:31:27.623170 20592 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_encrypt_loopback_connections=true --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:46652 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:27.637845 20592 master_main.cc:80] Initializing master server... I0627 19:31:27.641721 20592 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 212592us Jun 27 19:31:27 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:27.648236 20592 init.cc:339] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu) I0627 19:31:27.657796 20592 fs_manager.cc:261] Metadata directory not provided I0627 19:31:27.658030 20592 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:27.658435 20592 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:27.658658 20592 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:27.675004 20592 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "b5ac9e8f6a4b4c589e552ab519495585" format_stamp: "Formatted at 2018-06-28 02:31:27 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:27.676095 20592 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "b5ac9e8f6a4b4c589e552ab519495585" format_stamp: "Formatted at 2018-06-28 02:31:27 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:27.695552 20592 fs_manager.cc:501] Time spent creating directory manager: real 0.019s user 0.014s sys 0.003s I0627 19:31:27.697556 20592 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:27.699007 20592 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:27.716428 20592 fs_manager.cc:417] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s I0627 19:31:27.716682 20592 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/wal uuid: "b5ac9e8f6a4b4c589e552ab519495585" format_stamp: "Formatted at 2018-06-28 02:31:27 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:27.717046 20592 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:28.009536 20592 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:28.013312 20592 master_main.cc:83] Starting Master server... I0627 19:31:28.205533 20592 rpc_server.cc:205] RPC server started. Bound to: 127.0.0.1:46652 I0627 19:31:28.207011 20592 webserver.cc:175] Starting webserver on 127.0.0.1:0 I0627 19:31:28.207231 20592 webserver.cc:186] Document root disabled I0627 19:31:28.214160 20592 webserver.cc:313] Webserver started. Bound to: http://127.0.0.1:38184/ I0627 19:31:28.215360 20592 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:28.219662 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 20592 I0627 19:31:28.223619 20658 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed Jun 27 19:31:28 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM I0627 19:31:28.273097 20658 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585: Bootstrap starting. I0627 19:31:28.276793 20658 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585: No blocks or log segments found. Creating new log. I0627 19:31:28.279397 20658 log.cc:525] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585: Log is configured to *not* fsync() on all Append() calls I0627 19:31:28.290374 20658 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585: No bootstrap required, opened a new log I0627 19:31:28.292117 20660 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:28.223205 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:56809 0627 19:31:28.234794 (+ 11589us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:28.234832 (+ 38us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:28.234905 (+ 73us) server_negotiation.cc:373] Connection header received 0627 19:31:28.250441 (+ 15536us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:28.250496 (+ 55us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:28.250834 (+ 338us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:28.251206 (+ 372us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:28.255117 (+ 3911us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:28.255812 (+ 695us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:28.259026 (+ 3214us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:28.260936 (+ 1910us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:28.261169 (+ 233us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:28.272992 (+ 11823us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:28.273116 (+ 124us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:28.273153 (+ 37us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:28.273289 (+ 136us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:28.281996 (+ 8707us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:28.284392 (+ 2396us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:28.284455 (+ 63us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:28.284477 (+ 22us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:28.285184 (+ 707us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:28.286234 (+ 1050us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:28.286255 (+ 21us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:28.286273 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:28.286930 (+ 657us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:28.287479 (+ 549us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:28.288551 (+ 1072us) server_negotiation.cc:264] Negotiation successful 0627 19:31:28.289682 (+ 1131us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":11182,"thread_start_us":10978,"threads_started":1} I0627 19:31:28.295581 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:31:28.366243 20660 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:28.326289 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:40162 0627 19:31:28.326594 (+ 305us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:28.326615 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:28.334364 (+ 7749us) server_negotiation.cc:373] Connection header received 0627 19:31:28.334631 (+ 267us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:28.334656 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:28.334773 (+ 117us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:28.334889 (+ 116us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:28.343745 (+ 8856us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:28.344171 (+ 426us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:28.345580 (+ 1409us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:28.346587 (+ 1007us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:28.346762 (+ 175us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:28.349728 (+ 2966us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:28.349790 (+ 62us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:28.349813 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:28.349889 (+ 76us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:28.355995 (+ 6106us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:28.358681 (+ 2686us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:28.358708 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:28.358725 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:28.359509 (+ 784us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:28.360764 (+ 1255us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:28.360787 (+ 23us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:28.360804 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:28.361222 (+ 418us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:28.361676 (+ 454us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:28.364286 (+ 2610us) server_negotiation.cc:264] Negotiation successful 0627 19:31:28.364652 (+ 366us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":117} I0627 19:31:28.323765 20658 raft_consensus.cc:325] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac9e8f6a4b4c589e552ab519495585" member_type: VOTER } I0627 19:31:28.432945 20658 raft_consensus.cc:351] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:28.433104 20658 raft_consensus.cc:628] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ac9e8f6a4b4c589e552ab519495585, State: Initialized, Role: FOLLOWER I0627 19:31:28.433645 20658 consensus_queue.cc:227] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac9e8f6a4b4c589e552ab519495585" member_type: VOTER } I0627 19:31:28.433991 20658 raft_consensus.cc:365] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:28.434167 20658 raft_consensus.cc:457] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:28.434332 20658 raft_consensus.cc:2730] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585: Snoozing failure detection for 1.546s (starting election) I0627 19:31:28.434528 20658 raft_consensus.cc:2782] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:28.437292 20658 raft_consensus.cc:479] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac9e8f6a4b4c589e552ab519495585" member_type: VOTER } I0627 19:31:28.437767 20658 leader_election.cc:270] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:28.443783 20671 raft_consensus.cc:2730] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585: Snoozing failure detection for 1.697s (election complete) I0627 19:31:28.444161 20671 raft_consensus.cc:2537] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:28.453850 20671 raft_consensus.cc:593] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [term 1 LEADER]: Becoming Leader. State: Replica: b5ac9e8f6a4b4c589e552ab519495585, State: Running, Role: LEADER I0627 19:31:28.455976 20671 consensus_queue.cc:204] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac9e8f6a4b4c589e552ab519495585" member_type: VOTER } I0627 19:31:28.462638 20673 sys_catalog.cc:337] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ac9e8f6a4b4c589e552ab519495585. Latest consensus state: current_term: 1 leader_uuid: "b5ac9e8f6a4b4c589e552ab519495585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac9e8f6a4b4c589e552ab519495585" member_type: VOTER } } I0627 19:31:28.463223 20673 sys_catalog.cc:340] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [sys.catalog]: This master's current role is: LEADER I0627 19:31:28.463527 20672 sys_catalog.cc:337] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5ac9e8f6a4b4c589e552ab519495585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac9e8f6a4b4c589e552ab519495585" member_type: VOTER } } I0627 19:31:28.464215 20672 sys_catalog.cc:340] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [sys.catalog]: This master's current role is: LEADER I0627 19:31:28.464133 20658 sys_catalog.cc:337] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "b5ac9e8f6a4b4c589e552ab519495585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ac9e8f6a4b4c589e552ab519495585" member_type: VOTER } } I0627 19:31:28.464650 20658 sys_catalog.cc:340] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [sys.catalog]: This master's current role is: LEADER I0627 19:31:28.465557 20658 sys_catalog.cc:424] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:28.476986 20592 master_main.cc:86] Master server successfully started. W0627 19:31:28.485352 20676 catalog_manager.cc:1064] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:28.485769 20676 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:31:28.486053 20675 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:28.488914 20675 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... W0627 19:31:28.517742 20612 tablet.cc:1678] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:28.644959 20675 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:28.647225 20675 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:28.659579 20675 catalog_manager.cc:4102] T 00000000000000000000000000000000 P b5ac9e8f6a4b4c589e552ab519495585: Generated new TSK 0 I0627 19:31:28.689157 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 20592 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1836 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1 W0627 19:31:28.722703 20581 connection.cc:512] client connection to 127.0.0.1:46652 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46652): Connection reset by peer (error 104) Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:28 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20719](info): setting up network... Jun 27 19:31:28 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20719](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:28 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20719](info): skipping unrecognized local address family 17 Jun 27 19:31:28 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20719](info): skipping unrecognized local address family 17 Jun 27 19:31:28 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20719](info): set up 1 sockets Jun 27 19:31:28 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20719](info): commencing operation krb5kdc: starting... W0627 19:31:28.880942 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.131s user 0.002s sys 0.015s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:29 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.0.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. W0627 19:31:29.184082 12812 mini_kdc.cc:260] Time spent creating service keytab for kudu/127.0.0.1: real 0.101s user 0.000s sys 0.011s I0627 19:31:29.185444 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:42979 --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=required --rpc-encryption=required --rpc_encrypt_loopback_connections=false --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:29.424778 20758 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:29.425245 20758 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.0.0.1 W0627 19:31:29.425668 20758 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:29.459235 20758 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:29.459497 20758 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:29.459661 20758 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:29.459815 20758 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:29.459993 20758 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:29.460191 20758 flags.cc:406] Enabled experimental flag: --principal=kudu/127.0.0.1 I0627 19:31:29.512668 20758 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:42979 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:29.528080 20758 master_main.cc:80] Initializing master server... I0627 19:31:29.531896 20758 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 213592us Jun 27 19:31:29 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:29.540387 20758 init.cc:339] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu) I0627 19:31:29.562657 20758 fs_manager.cc:261] Metadata directory not provided I0627 19:31:29.562896 20758 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:29.563305 20758 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:29.563529 20758 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:29.579198 20758 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "f4a32726f78a4806b521bb5a4ced9de7" format_stamp: "Formatted at 2018-06-28 02:31:29 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:29.580577 20758 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "f4a32726f78a4806b521bb5a4ced9de7" format_stamp: "Formatted at 2018-06-28 02:31:29 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:29.603021 20758 fs_manager.cc:501] Time spent creating directory manager: real 0.022s user 0.010s sys 0.009s I0627 19:31:29.604936 20758 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:29.606700 20758 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:29.629492 20758 fs_manager.cc:417] Time spent opening block manager: real 0.011s user 0.001s sys 0.005s I0627 19:31:29.629770 20758 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/wal uuid: "f4a32726f78a4806b521bb5a4ced9de7" format_stamp: "Formatted at 2018-06-28 02:31:29 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:29.630206 20758 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:29.930600 20758 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:29.938577 20758 master_main.cc:83] Starting Master server... I0627 19:31:30.254029 20758 rpc_server.cc:205] RPC server started. Bound to: 127.0.0.1:42979 I0627 19:31:30.255349 20758 webserver.cc:175] Starting webserver on 127.0.0.1:0 I0627 19:31:30.255533 20758 webserver.cc:186] Document root disabled I0627 19:31:30.264907 20758 webserver.cc:313] Webserver started. Bound to: http://127.0.0.1:45916/ I0627 19:31:30.265982 20758 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:30.273916 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 20758 I0627 19:31:30.293849 20947 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed Jun 27 19:31:30 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[20719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM I0627 19:31:30.329638 20947 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7: Bootstrap starting. I0627 19:31:30.332707 20947 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7: No blocks or log segments found. Creating new log. I0627 19:31:30.335098 20947 log.cc:525] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7: Log is configured to *not* fsync() on all Append() calls I0627 19:31:30.340127 20953 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:30.282772 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:60398 0627 19:31:30.294445 (+ 11673us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:30.294482 (+ 37us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:30.294557 (+ 75us) server_negotiation.cc:373] Connection header received 0627 19:31:30.306736 (+ 12179us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:30.306801 (+ 65us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:30.307215 (+ 414us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:30.307637 (+ 422us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:30.310967 (+ 3330us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:30.311688 (+ 721us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:30.313279 (+ 1591us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:30.315165 (+ 1886us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:30.315411 (+ 246us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:30.319623 (+ 4212us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:30.319656 (+ 33us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:30.319687 (+ 31us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:30.319784 (+ 97us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:30.330512 (+ 10728us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:30.332363 (+ 1851us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:30.332414 (+ 51us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:30.332432 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:30.333173 (+ 741us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:30.333692 (+ 519us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:30.333716 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:30.333730 (+ 14us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:30.334427 (+ 697us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:30.334844 (+ 417us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:30.336658 (+ 1814us) server_negotiation.cc:264] Negotiation successful 0627 19:31:30.337978 (+ 1320us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":11180,"thread_start_us":10508,"threads_started":1} I0627 19:31:30.344537 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:31:30.353006 20947 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7: No bootstrap required, opened a new log I0627 19:31:30.382611 20947 raft_consensus.cc:325] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a32726f78a4806b521bb5a4ced9de7" member_type: VOTER } I0627 19:31:30.383195 20947 raft_consensus.cc:351] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:30.383456 20947 raft_consensus.cc:628] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a32726f78a4806b521bb5a4ced9de7, State: Initialized, Role: FOLLOWER I0627 19:31:30.384515 20947 consensus_queue.cc:227] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a32726f78a4806b521bb5a4ced9de7" member_type: VOTER } I0627 19:31:30.385108 20947 raft_consensus.cc:365] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:30.385344 20947 raft_consensus.cc:457] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:30.385656 20947 raft_consensus.cc:2730] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7: Snoozing failure detection for 1.878s (starting election) I0627 19:31:30.385907 20947 raft_consensus.cc:2782] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:30.390225 20947 raft_consensus.cc:479] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a32726f78a4806b521bb5a4ced9de7" member_type: VOTER } I0627 19:31:30.390903 20947 leader_election.cc:270] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:30.398196 20987 raft_consensus.cc:2730] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7: Snoozing failure detection for 1.547s (election complete) I0627 19:31:30.400957 20987 raft_consensus.cc:2537] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:30.404495 20953 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:30.370062 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:44010 0627 19:31:30.370662 (+ 600us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:30.370682 (+ 20us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:30.378985 (+ 8303us) server_negotiation.cc:373] Connection header received 0627 19:31:30.379417 (+ 432us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:30.379444 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:30.379571 (+ 127us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:30.379698 (+ 127us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:30.382110 (+ 2412us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:30.382532 (+ 422us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:30.384048 (+ 1516us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:30.385055 (+ 1007us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:30.385216 (+ 161us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:30.388277 (+ 3061us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:30.388302 (+ 25us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:30.388322 (+ 20us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:30.388423 (+ 101us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:30.395117 (+ 6694us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:30.397007 (+ 1890us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:30.397033 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:30.397050 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:30.397686 (+ 636us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:30.398229 (+ 543us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:30.398257 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:30.398274 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:30.398789 (+ 515us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:30.399071 (+ 282us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:30.400899 (+ 1828us) server_negotiation.cc:264] Negotiation successful 0627 19:31:30.401310 (+ 411us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":412} I0627 19:31:30.406690 20987 raft_consensus.cc:593] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [term 1 LEADER]: Becoming Leader. State: Replica: f4a32726f78a4806b521bb5a4ced9de7, State: Running, Role: LEADER I0627 19:31:30.573217 20987 consensus_queue.cc:204] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a32726f78a4806b521bb5a4ced9de7" member_type: VOTER } I0627 19:31:30.591162 20991 sys_catalog.cc:337] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a32726f78a4806b521bb5a4ced9de7. Latest consensus state: current_term: 1 leader_uuid: "f4a32726f78a4806b521bb5a4ced9de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a32726f78a4806b521bb5a4ced9de7" member_type: VOTER } } I0627 19:31:30.591717 20991 sys_catalog.cc:340] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [sys.catalog]: This master's current role is: LEADER I0627 19:31:30.593242 20947 sys_catalog.cc:337] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "f4a32726f78a4806b521bb5a4ced9de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a32726f78a4806b521bb5a4ced9de7" member_type: VOTER } } I0627 19:31:30.593677 20947 sys_catalog.cc:340] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [sys.catalog]: This master's current role is: LEADER I0627 19:31:30.594899 20947 sys_catalog.cc:424] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:30.592758 20989 sys_catalog.cc:337] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4a32726f78a4806b521bb5a4ced9de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a32726f78a4806b521bb5a4ced9de7" member_type: VOTER } } I0627 19:31:30.595471 20989 sys_catalog.cc:340] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7 [sys.catalog]: This master's current role is: LEADER I0627 19:31:30.609539 20758 master_main.cc:86] Master server successfully started. I0627 19:31:30.608799 21024 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:30.632681 21024 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... W0627 19:31:30.696216 20845 tablet.cc:1678] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:30.793459 21024 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:30.795720 21024 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:30.842761 21024 catalog_manager.cc:4102] T 00000000000000000000000000000000 P f4a32726f78a4806b521bb5a4ced9de7: Generated new TSK 0 I0627 19:31:30.854128 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 20758 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2193 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:31 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21062](info): setting up network... Jun 27 19:31:31 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21062](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:31 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21062](info): skipping unrecognized local address family 17 Jun 27 19:31:31 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21062](info): skipping unrecognized local address family 17 Jun 27 19:31:31 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21062](info): set up 1 sockets Jun 27 19:31:31 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21062](info): commencing operation krb5kdc: starting... W0627 19:31:31.255170 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.314s user 0.002s sys 0.016s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:31 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.0.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. I0627 19:31:31.621793 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:55934 --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=required --rpc_encrypt_loopback_connections=true --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:31.829439 21107 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:31.829926 21107 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.0.0.1 W0627 19:31:31.830338 21107 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:31.846853 21107 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:31.847671 21107 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:31.847800 21107 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:31.847919 21107 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:31.848052 21107 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:31.848191 21107 flags.cc:406] Enabled experimental flag: --principal=kudu/127.0.0.1 I0627 19:31:31.897089 21107 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_encrypt_loopback_connections=true --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:55934 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=disabled --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:31.911759 21107 master_main.cc:80] Initializing master server... I0627 19:31:31.914181 21107 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 214592us Jun 27 19:31:31 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:31.917315 21107 init.cc:339] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu) I0627 19:31:31.930711 21107 fs_manager.cc:261] Metadata directory not provided I0627 19:31:31.930948 21107 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:31.931357 21107 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:31.931569 21107 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:31.946348 21107 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" format_stamp: "Formatted at 2018-06-28 02:31:31 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:31.947480 21107 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" format_stamp: "Formatted at 2018-06-28 02:31:31 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:31.967787 21107 fs_manager.cc:501] Time spent creating directory manager: real 0.020s user 0.011s sys 0.007s I0627 19:31:31.969666 21107 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:31.970294 21107 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:31.985241 21107 fs_manager.cc:417] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s I0627 19:31:31.985510 21107 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/wal uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" format_stamp: "Formatted at 2018-06-28 02:31:31 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:31.985877 21107 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:32.279542 21107 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:32.284610 21107 master_main.cc:83] Starting Master server... I0627 19:31:32.647402 21107 rpc_server.cc:205] RPC server started. Bound to: 127.0.0.1:55934 I0627 19:31:32.648865 21107 webserver.cc:175] Starting webserver on 127.0.0.1:0 I0627 19:31:32.649045 21107 webserver.cc:186] Document root disabled I0627 19:31:32.663736 21107 webserver.cc:313] Webserver started. Bound to: http://127.0.0.1:35153/ I0627 19:31:32.664541 21107 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:32.673926 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 21107 I0627 19:31:32.685442 21385 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:32.730206 21385 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3: Bootstrap starting. I0627 19:31:32.731387 21390 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:32.676455 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:33081 0627 19:31:32.682761 (+ 6306us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:32.682797 (+ 36us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:32.682866 (+ 69us) server_negotiation.cc:373] Connection header received 0627 19:31:32.698245 (+ 15379us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:32.698296 (+ 51us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:32.698599 (+ 303us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:32.700312 (+ 1713us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:32.706787 (+ 6475us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:32.707591 (+ 804us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:32.709509 (+ 1918us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:32.713487 (+ 3978us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:32.713699 (+ 212us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:32.718475 (+ 4776us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:32.718565 (+ 90us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:32.718598 (+ 33us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:32.719190 (+ 592us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:32.719302 (+ 112us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:32.719403 (+ 101us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:32.719720 (+ 317us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:32.721036 (+ 1316us) server_negotiation.cc:264] Negotiation successful 0627 19:31:32.721176 (+ 140us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5884,"thread_start_us":5249,"threads_started":1} I0627 19:31:32.734830 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:31:32.736208 21385 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3: No blocks or log segments found. Creating new log. I0627 19:31:32.738467 21385 log.cc:525] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3: Log is configured to *not* fsync() on all Append() calls I0627 19:31:32.745863 21385 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3: No bootstrap required, opened a new log I0627 19:31:32.764498 21385 raft_consensus.cc:325] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" member_type: VOTER } I0627 19:31:32.766535 21385 raft_consensus.cc:351] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:32.766772 21385 raft_consensus.cc:628] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca07d3e0d0db4f268eeebf2b940f30d3, State: Initialized, Role: FOLLOWER I0627 19:31:32.767551 21385 consensus_queue.cc:227] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" member_type: VOTER } I0627 19:31:32.768067 21385 raft_consensus.cc:365] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:32.768319 21385 raft_consensus.cc:457] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:32.768604 21385 raft_consensus.cc:2730] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3: Snoozing failure detection for 2.048s (starting election) I0627 19:31:32.768849 21385 raft_consensus.cc:2782] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:32.774893 21385 raft_consensus.cc:479] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" member_type: VOTER } I0627 19:31:32.776510 21385 leader_election.cc:270] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:32.799815 21405 raft_consensus.cc:2730] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3: Snoozing failure detection for 2.149s (election complete) I0627 19:31:32.800186 21405 raft_consensus.cc:2537] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:32.803252 21390 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:32.781736 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:49197 0627 19:31:32.782083 (+ 347us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:32.782105 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:32.783783 (+ 1678us) server_negotiation.cc:373] Connection header received 0627 19:31:32.784011 (+ 228us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:32.784035 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:32.784146 (+ 111us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:32.784264 (+ 118us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:32.787245 (+ 2981us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:32.787681 (+ 436us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:32.797941 (+ 10260us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:32.799128 (+ 1187us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:32.799292 (+ 164us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:32.800762 (+ 1470us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:32.800824 (+ 62us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:32.800845 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:32.800894 (+ 49us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:32.800961 (+ 67us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:32.801021 (+ 60us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:32.801256 (+ 235us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:32.801836 (+ 580us) server_negotiation.cc:264] Negotiation successful 0627 19:31:32.801918 (+ 82us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":148} I0627 19:31:32.810951 21405 raft_consensus.cc:593] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [term 1 LEADER]: Becoming Leader. State: Replica: ca07d3e0d0db4f268eeebf2b940f30d3, State: Running, Role: LEADER I0627 19:31:33.010522 21405 consensus_queue.cc:204] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" member_type: VOTER } I0627 19:31:33.016839 21406 sys_catalog.cc:337] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca07d3e0d0db4f268eeebf2b940f30d3. Latest consensus state: current_term: 1 leader_uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" member_type: VOTER } } I0627 19:31:33.017041 21407 sys_catalog.cc:337] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" member_type: VOTER } } I0627 19:31:33.017364 21406 sys_catalog.cc:340] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [sys.catalog]: This master's current role is: LEADER I0627 19:31:33.017460 21407 sys_catalog.cc:340] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [sys.catalog]: This master's current role is: LEADER I0627 19:31:33.017788 21385 sys_catalog.cc:337] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca07d3e0d0db4f268eeebf2b940f30d3" member_type: VOTER } } I0627 19:31:33.018268 21385 sys_catalog.cc:340] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [sys.catalog]: This master's current role is: LEADER I0627 19:31:33.019836 21385 sys_catalog.cc:424] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:33.036720 21423 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0627 19:31:33.037854 21107 master_main.cc:86] Master server successfully started. I0627 19:31:33.038210 21422 catalog_manager.cc:1006] Loading table and tablet metadata into memory... W0627 19:31:33.048108 21297 tablet.cc:1678] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:33.048058 21422 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0627 19:31:33.229754 21422 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:33.232136 21422 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:33.250640 21422 catalog_manager.cc:4102] T 00000000000000000000000000000000 P ca07d3e0d0db4f268eeebf2b940f30d3: Generated new TSK 0 I0627 19:31:33.274931 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 21107 W0627 19:31:33.337085 21045 connection.cc:512] client connection to 127.0.0.1:55934 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:55934): Connection reset by peer (error 104) [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2419 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:33 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21469](info): setting up network... Jun 27 19:31:33 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21469](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:33 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21469](info): skipping unrecognized local address family 17 Jun 27 19:31:33 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21469](info): skipping unrecognized local address family 17 Jun 27 19:31:33 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21469](info): set up 1 sockets Jun 27 19:31:33 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21469](info): commencing operation krb5kdc: starting... W0627 19:31:33.561640 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.208s user 0.001s sys 0.016s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:33 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.0.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. I0627 19:31:33.852267 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:51073 --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=required --rpc_encrypt_loopback_connections=false --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:34.120409 21513 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:34.120935 21513 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.0.0.1 W0627 19:31:34.121354 21513 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:34.142148 21513 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:34.142536 21513 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:34.142741 21513 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:34.142937 21513 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:34.143158 21513 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:34.143399 21513 flags.cc:406] Enabled experimental flag: --principal=kudu/127.0.0.1 I0627 19:31:34.200959 21513 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:51073 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=disabled --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:34.215551 21513 master_main.cc:80] Initializing master server... I0627 19:31:34.217991 21513 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 216092us Jun 27 19:31:34 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:34.222690 21513 init.cc:339] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu) I0627 19:31:34.234858 21513 fs_manager.cc:261] Metadata directory not provided I0627 19:31:34.235110 21513 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:34.235554 21513 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:34.235750 21513 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:34.251139 21513 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" format_stamp: "Formatted at 2018-06-28 02:31:34 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:34.252707 21513 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" format_stamp: "Formatted at 2018-06-28 02:31:34 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:34.272624 21513 fs_manager.cc:501] Time spent creating directory manager: real 0.019s user 0.013s sys 0.003s I0627 19:31:34.274515 21513 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:34.275202 21513 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:34.292557 21513 fs_manager.cc:417] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s I0627 19:31:34.292795 21513 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/wal uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" format_stamp: "Formatted at 2018-06-28 02:31:34 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:34.293112 21513 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:34.605818 21513 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:34.615273 21513 master_main.cc:83] Starting Master server... I0627 19:31:34.924744 21513 rpc_server.cc:205] RPC server started. Bound to: 127.0.0.1:51073 I0627 19:31:34.926452 21513 webserver.cc:175] Starting webserver on 127.0.0.1:0 I0627 19:31:34.926635 21513 webserver.cc:186] Document root disabled I0627 19:31:34.934499 21513 webserver.cc:313] Webserver started. Bound to: http://127.0.0.1:40294/ I0627 19:31:34.935600 21513 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:34.942121 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 21513 I0627 19:31:34.967020 21600 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:35.034086 21600 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6: Bootstrap starting. I0627 19:31:35.037627 21602 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:34.944004 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:42988 0627 19:31:34.972514 (+ 28510us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:34.972555 (+ 41us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:34.972634 (+ 79us) server_negotiation.cc:373] Connection header received 0627 19:31:35.014477 (+ 41843us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:35.014530 (+ 53us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:35.014883 (+ 353us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:35.016533 (+ 1650us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:35.016928 (+ 395us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:35.017681 (+ 753us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:35.020986 (+ 3305us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:35.028970 (+ 7984us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:35.030290 (+ 1320us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:35.030759 (+ 469us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:35.030797 (+ 38us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:35.030827 (+ 30us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:35.031536 (+ 709us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:35.031701 (+ 165us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:35.031781 (+ 80us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:35.035301 (+ 3520us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:35.035466 (+ 165us) server_negotiation.cc:264] Negotiation successful 0627 19:31:35.035714 (+ 248us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":8885,"thread_start_us":7483,"threads_started":1} I0627 19:31:35.050792 21600 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6: No blocks or log segments found. Creating new log. I0627 19:31:35.051017 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:31:35.052722 21600 log.cc:525] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6: Log is configured to *not* fsync() on all Append() calls I0627 19:31:35.068928 21600 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6: No bootstrap required, opened a new log I0627 19:31:35.110013 21600 raft_consensus.cc:325] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" member_type: VOTER } I0627 19:31:35.111477 21600 raft_consensus.cc:351] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:35.111744 21600 raft_consensus.cc:628] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341e0c2bd4fa4b11b9388c8b501c31e6, State: Initialized, Role: FOLLOWER I0627 19:31:35.112491 21600 consensus_queue.cc:227] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" member_type: VOTER } I0627 19:31:35.117321 21600 raft_consensus.cc:365] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:35.117951 21600 raft_consensus.cc:457] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:35.118532 21600 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6: Snoozing failure detection for 2.000s (starting election) I0627 19:31:35.122570 21600 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:35.121824 21602 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:35.100444 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:54392 0627 19:31:35.100771 (+ 327us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:35.100792 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:35.108570 (+ 7778us) server_negotiation.cc:373] Connection header received 0627 19:31:35.108984 (+ 414us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:35.109034 (+ 50us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:35.109172 (+ 138us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:35.109309 (+ 137us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:35.110639 (+ 1330us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:35.111030 (+ 391us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:35.116822 (+ 5792us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:35.117914 (+ 1092us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:35.118097 (+ 183us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:35.119446 (+ 1349us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:35.119475 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:35.119498 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:35.119551 (+ 53us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:35.119626 (+ 75us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:35.119697 (+ 71us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:35.119797 (+ 100us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:35.120199 (+ 402us) server_negotiation.cc:264] Negotiation successful 0627 19:31:35.120408 (+ 209us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":83} I0627 19:31:35.136389 21600 raft_consensus.cc:479] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" member_type: VOTER } I0627 19:31:35.137370 21600 leader_election.cc:270] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:35.144806 21616 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6: Snoozing failure detection for 2.056s (election complete) I0627 19:31:35.145305 21616 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:35.156600 21616 raft_consensus.cc:593] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [term 1 LEADER]: Becoming Leader. State: Replica: 341e0c2bd4fa4b11b9388c8b501c31e6, State: Running, Role: LEADER I0627 19:31:35.158004 21616 consensus_queue.cc:204] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" member_type: VOTER } I0627 19:31:35.176818 21616 sys_catalog.cc:337] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" member_type: VOTER } } I0627 19:31:35.177372 21616 sys_catalog.cc:340] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [sys.catalog]: This master's current role is: LEADER I0627 19:31:35.195242 21600 sys_catalog.cc:337] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" member_type: VOTER } } I0627 19:31:35.381023 21600 sys_catalog.cc:340] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [sys.catalog]: This master's current role is: LEADER I0627 19:31:35.382576 21600 sys_catalog.cc:424] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:35.195909 21618 sys_catalog.cc:337] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 341e0c2bd4fa4b11b9388c8b501c31e6. Latest consensus state: current_term: 1 leader_uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341e0c2bd4fa4b11b9388c8b501c31e6" member_type: VOTER } } I0627 19:31:35.393394 21618 sys_catalog.cc:340] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6 [sys.catalog]: This master's current role is: LEADER I0627 19:31:35.394629 21623 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:35.395467 21513 master_main.cc:86] Master server successfully started. I0627 19:31:35.414465 21623 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0627 19:31:35.526569 21623 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:35.529000 21623 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:35.551100 21623 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 341e0c2bd4fa4b11b9388c8b501c31e6: Generated new TSK 0 I0627 19:31:35.576663 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 21513 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2294 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:35 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21630](info): setting up network... Jun 27 19:31:35 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21630](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:35 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21630](info): skipping unrecognized local address family 17 Jun 27 19:31:35 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21630](info): skipping unrecognized local address family 17 Jun 27 19:31:35 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21630](info): set up 1 sockets Jun 27 19:31:35 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21630](info): commencing operation krb5kdc: starting... W0627 19:31:36.078333 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.410s user 0.003s sys 0.014s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:36 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.0.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. W0627 19:31:36.536645 12812 mini_kdc.cc:260] Time spent creating service keytab for kudu/127.0.0.1: real 0.106s user 0.001s sys 0.010s I0627 19:31:36.538152 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:45974 --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=disabled --rpc_encrypt_loopback_connections=true --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:36.952836 21640 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:36.953371 21640 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.0.0.1 W0627 19:31:36.953833 21640 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:37.001540 21640 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:37.008664 21640 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:37.008909 21640 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:37.009105 21640 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:37.009335 21640 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:37.009609 21640 flags.cc:406] Enabled experimental flag: --principal=kudu/127.0.0.1 I0627 19:31:37.181510 21640 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_encrypt_loopback_connections=true --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:45974 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=disabled --rpc_encryption=disabled --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:37.221555 21640 master_main.cc:80] Initializing master server... I0627 19:31:37.256986 21640 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 217592us Jun 27 19:31:37 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:37.266588 21640 init.cc:339] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu) I0627 19:31:37.291023 21640 fs_manager.cc:261] Metadata directory not provided I0627 19:31:37.291214 21640 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:37.291627 21640 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:37.291796 21640 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:37.329977 21640 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" format_stamp: "Formatted at 2018-06-28 02:31:37 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:37.335693 21640 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" format_stamp: "Formatted at 2018-06-28 02:31:37 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:37.380378 21640 fs_manager.cc:501] Time spent creating directory manager: real 0.043s user 0.015s sys 0.000s I0627 19:31:37.382419 21640 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:37.383095 21640 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:37.429963 21640 fs_manager.cc:417] Time spent opening block manager: real 0.027s user 0.002s sys 0.004s I0627 19:31:37.430155 21640 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/wal uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" format_stamp: "Formatted at 2018-06-28 02:31:37 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:37.430516 21640 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:37.989660 21640 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:38.026531 21640 master_main.cc:83] Starting Master server... I0627 19:31:38.313993 21640 rpc_server.cc:205] RPC server started. Bound to: 127.0.0.1:45974 I0627 19:31:38.315112 21640 webserver.cc:175] Starting webserver on 127.0.0.1:0 I0627 19:31:38.315238 21640 webserver.cc:186] Document root disabled I0627 19:31:38.317688 21640 webserver.cc:313] Webserver started. Bound to: http://127.0.0.1:50208/ I0627 19:31:38.324710 21640 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:38.326153 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 21640 I0627 19:31:38.343228 21705 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:38.367115 21704 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:38.329843 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:36471 0627 19:31:38.332001 (+ 2158us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:38.332045 (+ 44us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:38.332127 (+ 82us) server_negotiation.cc:373] Connection header received 0627 19:31:38.354520 (+ 22393us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:38.354575 (+ 55us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:38.354851 (+ 276us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:38.361840 (+ 6989us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:38.362844 (+ 1004us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:38.362881 (+ 37us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:38.362912 (+ 31us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:38.362968 (+ 56us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:38.363089 (+ 121us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:38.363173 (+ 84us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:38.364061 (+ 888us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:38.364267 (+ 206us) server_negotiation.cc:264] Negotiation successful 0627 19:31:38.364446 (+ 179us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1723,"thread_start_us":1237,"threads_started":1} I0627 19:31:38.371345 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:31:38.383994 21705 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0: Bootstrap starting. I0627 19:31:38.389529 21705 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0: No blocks or log segments found. Creating new log. I0627 19:31:38.390817 21705 log.cc:525] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0: Log is configured to *not* fsync() on all Append() calls I0627 19:31:38.406847 21705 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0: No bootstrap required, opened a new log I0627 19:31:38.445118 21705 raft_consensus.cc:325] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" member_type: VOTER } I0627 19:31:38.445766 21705 raft_consensus.cc:351] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:38.446022 21705 raft_consensus.cc:628] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17bb3ab37e747ee83a82d634d1fa2e0, State: Initialized, Role: FOLLOWER I0627 19:31:38.446895 21705 consensus_queue.cc:227] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" member_type: VOTER } I0627 19:31:38.447495 21705 raft_consensus.cc:365] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:38.447785 21705 raft_consensus.cc:457] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:38.448086 21705 raft_consensus.cc:2730] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0: Snoozing failure detection for 2.248s (starting election) I0627 19:31:38.448374 21705 raft_consensus.cc:2782] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:38.453056 21705 raft_consensus.cc:479] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" member_type: VOTER } I0627 19:31:38.453858 21705 leader_election.cc:270] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:38.461285 21712 raft_consensus.cc:2730] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0: Snoozing failure detection for 2.149s (election complete) I0627 19:31:38.463392 21712 raft_consensus.cc:2537] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:38.463703 21704 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:38.430568 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:42264 0627 19:31:38.431223 (+ 655us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:38.431246 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:38.451344 (+ 20098us) server_negotiation.cc:373] Connection header received 0627 19:31:38.451677 (+ 333us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:38.451706 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:38.451817 (+ 111us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:38.451938 (+ 121us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:38.458562 (+ 6624us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:38.458593 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:38.458615 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:38.458666 (+ 51us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:38.458746 (+ 80us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:38.458805 (+ 59us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:38.458913 (+ 108us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:38.462155 (+ 3242us) server_negotiation.cc:264] Negotiation successful 0627 19:31:38.462253 (+ 98us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":449} I0627 19:31:38.467559 21712 raft_consensus.cc:593] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [term 1 LEADER]: Becoming Leader. State: Replica: b17bb3ab37e747ee83a82d634d1fa2e0, State: Running, Role: LEADER I0627 19:31:38.468052 21712 consensus_queue.cc:204] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" member_type: VOTER } I0627 19:31:38.496353 21713 sys_catalog.cc:337] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b17bb3ab37e747ee83a82d634d1fa2e0. Latest consensus state: current_term: 1 leader_uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" member_type: VOTER } } I0627 19:31:38.496893 21713 sys_catalog.cc:340] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [sys.catalog]: This master's current role is: LEADER I0627 19:31:38.496626 21714 sys_catalog.cc:337] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" member_type: VOTER } } I0627 19:31:38.747617 21714 sys_catalog.cc:340] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [sys.catalog]: This master's current role is: LEADER I0627 19:31:38.497694 21705 sys_catalog.cc:337] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17bb3ab37e747ee83a82d634d1fa2e0" member_type: VOTER } } I0627 19:31:38.766924 21705 sys_catalog.cc:340] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [sys.catalog]: This master's current role is: LEADER I0627 19:31:38.768610 21705 sys_catalog.cc:424] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:38.780618 21717 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:38.787984 21640 master_main.cc:86] Master server successfully started. W0627 19:31:38.797202 21659 tablet.cc:1678] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:38.801620 21717 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0627 19:31:39.013972 21717 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:39.014506 21717 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:39.030858 21717 catalog_manager.cc:4102] T 00000000000000000000000000000000 P b17bb3ab37e747ee83a82d634d1fa2e0: Generated new TSK 0 I0627 19:31:39.051297 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 21640 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (3428 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:39 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21739](info): setting up network... Jun 27 19:31:39 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21739](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:39 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21739](info): skipping unrecognized local address family 17 Jun 27 19:31:39 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21739](info): skipping unrecognized local address family 17 Jun 27 19:31:39 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21739](info): set up 1 sockets Jun 27 19:31:39 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21739](info): commencing operation krb5kdc: starting... W0627 19:31:39.248172 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.163s user 0.001s sys 0.016s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:39 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.0.0.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. Entry for principal kudu/127.0.0.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab. I0627 19:31:39.478463 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:49252 --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=disabled --rpc_encrypt_loopback_connections=false --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:39.702533 21798 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:39.703029 21798 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.0.0.1 W0627 19:31:39.703471 21798 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:39.739107 21798 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:39.739423 21798 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:39.739617 21798 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:39.739819 21798 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:39.740038 21798 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:39.740276 21798 flags.cc:406] Enabled experimental flag: --principal=kudu/127.0.0.1 I0627 19:31:39.763193 21798 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.0.1:49252 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/krb5kdc/kudu_127.0.0.1.keytab --principal=kudu/127.0.0.1 --rpc_authentication=disabled --rpc_encryption=disabled --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.0.0.1 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:39.771844 21798 master_main.cc:80] Initializing master server... I0627 19:31:39.773452 21798 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 218592us Jun 27 19:31:39 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:39.775717 21798 init.cc:339] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu) I0627 19:31:39.779247 21798 fs_manager.cc:261] Metadata directory not provided I0627 19:31:39.779384 21798 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:39.779696 21798 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:39.779817 21798 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:39.792675 21798 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "7ff6b80d87c747cfbc6991e0699e4013" format_stamp: "Formatted at 2018-06-28 02:31:39 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:39.793768 21798 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "7ff6b80d87c747cfbc6991e0699e4013" format_stamp: "Formatted at 2018-06-28 02:31:39 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:39.807297 21798 fs_manager.cc:501] Time spent creating directory manager: real 0.013s user 0.009s sys 0.003s I0627 19:31:39.808753 21798 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:39.809237 21798 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:39.817261 21798 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0627 19:31:39.817405 21798 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/wal uuid: "7ff6b80d87c747cfbc6991e0699e4013" format_stamp: "Formatted at 2018-06-28 02:31:39 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:39.817664 21798 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:40.002012 21798 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:40.005625 21798 master_main.cc:83] Starting Master server... I0627 19:31:40.141484 21798 rpc_server.cc:205] RPC server started. Bound to: 127.0.0.1:49252 I0627 19:31:40.142593 21798 webserver.cc:175] Starting webserver on 127.0.0.1:0 I0627 19:31:40.142711 21798 webserver.cc:186] Document root disabled I0627 19:31:40.145052 21798 webserver.cc:313] Webserver started. Bound to: http://127.0.0.1:50343/ I0627 19:31:40.145823 21798 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:40.149847 21929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:40.152212 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 21798 I0627 19:31:40.173689 21931 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:40.154756 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:47349 0627 19:31:40.163903 (+ 9147us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:40.163949 (+ 46us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:40.164023 (+ 74us) server_negotiation.cc:373] Connection header received 0627 19:31:40.168402 (+ 4379us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:40.168453 (+ 51us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:40.168773 (+ 320us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:40.169171 (+ 398us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:40.170855 (+ 1684us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:40.170887 (+ 32us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:40.170914 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:40.170963 (+ 49us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:40.171087 (+ 124us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:40.171165 (+ 78us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:40.171290 (+ 125us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:40.171658 (+ 368us) server_negotiation.cc:264] Negotiation successful 0627 19:31:40.171828 (+ 170us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":8692,"thread_start_us":6930,"threads_started":1} I0627 19:31:40.182199 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:31:40.183881 21929 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013: Bootstrap starting. I0627 19:31:40.186646 21929 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013: No blocks or log segments found. Creating new log. I0627 19:31:40.188261 21929 log.cc:525] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013: Log is configured to *not* fsync() on all Append() calls I0627 19:31:40.193035 21929 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013: No bootstrap required, opened a new log I0627 19:31:40.201263 21931 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:40.197486 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:41571 0627 19:31:40.197844 (+ 358us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:40.197866 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:40.198410 (+ 544us) server_negotiation.cc:373] Connection header received 0627 19:31:40.198531 (+ 121us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:40.198553 (+ 22us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:40.198662 (+ 109us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:40.198803 (+ 141us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:40.199413 (+ 610us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:40.199441 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:40.199463 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:40.199505 (+ 42us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:40.199585 (+ 80us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:40.199650 (+ 65us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:40.199766 (+ 116us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:40.200048 (+ 282us) server_negotiation.cc:264] Negotiation successful 0627 19:31:40.200132 (+ 84us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":164} I0627 19:31:40.207496 21929 raft_consensus.cc:325] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6b80d87c747cfbc6991e0699e4013" member_type: VOTER } I0627 19:31:40.294193 21929 raft_consensus.cc:351] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:40.294378 21929 raft_consensus.cc:628] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff6b80d87c747cfbc6991e0699e4013, State: Initialized, Role: FOLLOWER I0627 19:31:40.294906 21929 consensus_queue.cc:227] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6b80d87c747cfbc6991e0699e4013" member_type: VOTER } I0627 19:31:40.295294 21929 raft_consensus.cc:365] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:40.295483 21929 raft_consensus.cc:457] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:40.295656 21929 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013: Snoozing failure detection for 2.023s (starting election) I0627 19:31:40.295823 21929 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:40.298385 21929 raft_consensus.cc:479] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6b80d87c747cfbc6991e0699e4013" member_type: VOTER } I0627 19:31:40.298844 21929 leader_election.cc:270] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:40.304903 21938 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013: Snoozing failure detection for 1.796s (election complete) I0627 19:31:40.305300 21938 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:40.324048 21938 raft_consensus.cc:593] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [term 1 LEADER]: Becoming Leader. State: Replica: 7ff6b80d87c747cfbc6991e0699e4013, State: Running, Role: LEADER I0627 19:31:40.324512 21938 consensus_queue.cc:204] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6b80d87c747cfbc6991e0699e4013" member_type: VOTER } I0627 19:31:40.328042 21939 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ff6b80d87c747cfbc6991e0699e4013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6b80d87c747cfbc6991e0699e4013" member_type: VOTER } } I0627 19:31:40.328503 21939 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [sys.catalog]: This master's current role is: LEADER I0627 19:31:40.328752 21929 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "7ff6b80d87c747cfbc6991e0699e4013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6b80d87c747cfbc6991e0699e4013" member_type: VOTER } } I0627 19:31:40.329208 21929 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [sys.catalog]: This master's current role is: LEADER I0627 19:31:40.330515 21929 sys_catalog.cc:424] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:40.332164 21940 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ff6b80d87c747cfbc6991e0699e4013. Latest consensus state: current_term: 1 leader_uuid: "7ff6b80d87c747cfbc6991e0699e4013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff6b80d87c747cfbc6991e0699e4013" member_type: VOTER } } I0627 19:31:40.332633 21940 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013 [sys.catalog]: This master's current role is: LEADER I0627 19:31:40.343961 21798 master_main.cc:86] Master server successfully started. W0627 19:31:40.349267 21943 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:40.349580 21943 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:31:40.350000 21942 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:40.353478 21942 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0627 19:31:40.404637 21942 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:40.405186 21942 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:40.422942 21942 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 7ff6b80d87c747cfbc6991e0699e4013: Generated new TSK 0 I0627 19:31:40.451459 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 21798 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1402 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:40 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21949](info): setting up network... Jun 27 19:31:40 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21949](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:40 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21949](info): skipping unrecognized local address family 17 Jun 27 19:31:40 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21949](info): skipping unrecognized local address family 17 Jun 27 19:31:40 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21949](info): set up 1 sockets Jun 27 19:31:40 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21949](info): commencing operation krb5kdc: starting... W0627 19:31:40.604030 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.117s user 0.004s sys 0.011s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:40 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:31:40.800324 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:46814 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=required --rpc-encryption=required --rpc_encrypt_loopback_connections=true --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:40.984071 21959 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:40.984885 21959 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:31:40.985282 21959 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:41.002117 21959 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:41.002354 21959 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:41.002547 21959 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:41.002707 21959 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:41.002900 21959 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:41.003103 21959 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:31:41.039768 21959 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_encrypt_loopback_connections=true --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:46814 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:41.054051 21959 master_main.cc:80] Initializing master server... I0627 19:31:41.057104 21959 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 219592us Jun 27 19:31:41 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153101, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:41.063484 21959 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:31:41.073441 21959 fs_manager.cc:261] Metadata directory not provided I0627 19:31:41.073601 21959 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:41.073894 21959 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:41.074019 21959 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:41.083240 21959 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" format_stamp: "Formatted at 2018-06-28 02:31:41 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:41.084081 21959 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" format_stamp: "Formatted at 2018-06-28 02:31:41 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:41.106806 21959 fs_manager.cc:501] Time spent creating directory manager: real 0.022s user 0.009s sys 0.011s I0627 19:31:41.109124 21959 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:41.110604 21959 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:41.127929 21959 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.001s sys 0.005s I0627 19:31:41.128162 21959 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/wal uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" format_stamp: "Formatted at 2018-06-28 02:31:41 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:41.128535 21959 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:41.384723 21959 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:41.388371 21959 master_main.cc:83] Starting Master server... I0627 19:31:41.653120 21959 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:46814 I0627 19:31:41.654373 21959 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:31:41.654569 21959 webserver.cc:186] Document root disabled I0627 19:31:41.657477 21959 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:55836/ I0627 19:31:41.658691 21959 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:41.659178 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 21959 I0627 19:31:41.662452 22062 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed Jun 27 19:31:41 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[21949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:41.708398 22062 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e: Bootstrap starting. I0627 19:31:41.712672 22062 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e: No blocks or log segments found. Creating new log. I0627 19:31:41.717821 22062 log.cc:525] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e: Log is configured to *not* fsync() on all Append() calls I0627 19:31:41.729560 22062 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e: No bootstrap required, opened a new log I0627 19:31:41.733650 22065 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:41.666002 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:37020 0627 19:31:41.679184 (+ 13182us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:41.679218 (+ 34us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:41.679287 (+ 69us) server_negotiation.cc:373] Connection header received 0627 19:31:41.685149 (+ 5862us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:41.685200 (+ 51us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:41.685498 (+ 298us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:41.685854 (+ 356us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:41.690598 (+ 4744us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:41.691175 (+ 577us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:41.696639 (+ 5464us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:41.698327 (+ 1688us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:41.701421 (+ 3094us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:41.703646 (+ 2225us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:41.703733 (+ 87us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:41.703764 (+ 31us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:41.703844 (+ 80us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:41.719121 (+ 15277us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:41.726314 (+ 7193us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:41.726343 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:41.726363 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:41.727145 (+ 782us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:41.728192 (+ 1047us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:41.728208 (+ 16us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:41.728219 (+ 11us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:41.728916 (+ 697us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:41.729533 (+ 617us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:41.730778 (+ 1245us) server_negotiation.cc:264] Negotiation successful 0627 19:31:41.731199 (+ 421us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":12816,"thread_start_us":12146,"threads_started":1} I0627 19:31:41.745409 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:31:41.803632 22065 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:41.772231 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:33232 0627 19:31:41.772583 (+ 352us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:41.772609 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:41.773656 (+ 1047us) server_negotiation.cc:373] Connection header received 0627 19:31:41.773898 (+ 242us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:41.773924 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:41.774042 (+ 118us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:41.774160 (+ 118us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:41.776948 (+ 2788us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:41.777391 (+ 443us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:41.778800 (+ 1409us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:41.779795 (+ 995us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:41.779966 (+ 171us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:41.783064 (+ 3098us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:41.783130 (+ 66us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:41.783159 (+ 29us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:41.783246 (+ 87us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:41.789748 (+ 6502us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:41.797841 (+ 8093us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:41.797874 (+ 33us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:41.797895 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:41.798652 (+ 757us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:41.799823 (+ 1171us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:41.799847 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:41.799864 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:41.800300 (+ 436us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:41.800757 (+ 457us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:41.801641 (+ 884us) server_negotiation.cc:264] Negotiation successful 0627 19:31:41.801963 (+ 322us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":133} I0627 19:31:41.754526 22062 raft_consensus.cc:325] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" member_type: VOTER } I0627 19:31:41.889551 22062 raft_consensus.cc:351] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:41.889804 22062 raft_consensus.cc:628] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b6f73bc0ae4a7c9a8b6c033868510e, State: Initialized, Role: FOLLOWER I0627 19:31:41.890547 22062 consensus_queue.cc:227] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" member_type: VOTER } I0627 19:31:41.890924 22062 raft_consensus.cc:365] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:41.891103 22062 raft_consensus.cc:457] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:41.891284 22062 raft_consensus.cc:2730] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e: Snoozing failure detection for 2.183s (starting election) I0627 19:31:41.891484 22062 raft_consensus.cc:2782] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:41.895925 22062 raft_consensus.cc:479] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" member_type: VOTER } I0627 19:31:41.896661 22062 leader_election.cc:270] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:41.904126 22111 raft_consensus.cc:2730] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e: Snoozing failure detection for 1.502s (election complete) I0627 19:31:41.905418 22111 raft_consensus.cc:2537] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:41.910897 22111 raft_consensus.cc:593] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [term 1 LEADER]: Becoming Leader. State: Replica: e9b6f73bc0ae4a7c9a8b6c033868510e, State: Running, Role: LEADER I0627 19:31:41.911777 22111 consensus_queue.cc:204] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" member_type: VOTER } I0627 19:31:41.921658 22113 sys_catalog.cc:337] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b6f73bc0ae4a7c9a8b6c033868510e. Latest consensus state: current_term: 1 leader_uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" member_type: VOTER } } I0627 19:31:41.922129 22113 sys_catalog.cc:340] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [sys.catalog]: This master's current role is: LEADER I0627 19:31:41.922035 22062 sys_catalog.cc:337] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" member_type: VOTER } } I0627 19:31:41.922747 22062 sys_catalog.cc:340] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [sys.catalog]: This master's current role is: LEADER I0627 19:31:41.922859 22114 sys_catalog.cc:337] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b6f73bc0ae4a7c9a8b6c033868510e" member_type: VOTER } } I0627 19:31:41.924289 22114 sys_catalog.cc:340] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [sys.catalog]: This master's current role is: LEADER I0627 19:31:41.924510 22062 sys_catalog.cc:424] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:41.950176 21959 master_main.cc:86] Master server successfully started. W0627 19:31:41.952944 22121 catalog_manager.cc:1064] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:41.953166 22121 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:31:41.953613 22119 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:41.957669 22119 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0627 19:31:42.076020 22119 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:42.078290 22119 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:42.089359 22119 catalog_manager.cc:4102] T 00000000000000000000000000000000 P e9b6f73bc0ae4a7c9a8b6c033868510e: Generated new TSK 0 I0627 19:31:42.104146 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 21959 W0627 19:31:42.140086 21947 connection.cc:512] client connection to 127.12.131.62:46814 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.62:46814): Connection reset by peer (error 104) [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1669 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22171](info): setting up network... Jun 27 19:31:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22171](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22171](info): skipping unrecognized local address family 17 Jun 27 19:31:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22171](info): skipping unrecognized local address family 17 Jun 27 19:31:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22171](info): set up 1 sockets Jun 27 19:31:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22171](info): commencing operation krb5kdc: starting... W0627 19:31:42.380192 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.220s user 0.004s sys 0.014s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:42 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:31:42.592413 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:34797 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=required --rpc-encryption=required --rpc_encrypt_loopback_connections=false --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:42.892601 22181 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:42.895215 22181 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:31:42.895690 22181 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:42.917412 22181 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:42.920035 22181 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:42.920265 22181 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:42.920486 22181 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:42.920708 22181 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:42.920948 22181 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:31:42.972355 22181 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:34797 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:42.991384 22181 master_main.cc:80] Initializing master server... I0627 19:31:42.995486 22181 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 220092us Jun 27 19:31:43 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153103, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:43.004922 22181 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:31:43.012436 22181 fs_manager.cc:261] Metadata directory not provided I0627 19:31:43.012687 22181 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:43.013406 22181 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:43.013609 22181 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:43.032953 22181 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "d7de302f094f4c39abb61c2df94cb27d" format_stamp: "Formatted at 2018-06-28 02:31:43 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:43.034169 22181 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "d7de302f094f4c39abb61c2df94cb27d" format_stamp: "Formatted at 2018-06-28 02:31:43 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:43.059144 22181 fs_manager.cc:501] Time spent creating directory manager: real 0.024s user 0.010s sys 0.007s I0627 19:31:43.061880 22181 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:43.063712 22181 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:43.082381 22181 fs_manager.cc:417] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s I0627 19:31:43.082648 22181 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/wal uuid: "d7de302f094f4c39abb61c2df94cb27d" format_stamp: "Formatted at 2018-06-28 02:31:43 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:43.083009 22181 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:43.426048 22181 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:43.431337 22181 master_main.cc:83] Starting Master server... I0627 19:31:43.611857 22181 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:34797 I0627 19:31:43.613674 22181 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:31:43.613852 22181 webserver.cc:186] Document root disabled I0627 19:31:43.617825 22181 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:46968/ I0627 19:31:43.618788 22181 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:43.623467 22250 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:43.624308 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 22181 Jun 27 19:31:43 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:43.667179 22250 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d: Bootstrap starting. I0627 19:31:43.670526 22250 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d: No blocks or log segments found. Creating new log. I0627 19:31:43.672850 22250 log.cc:525] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d: Log is configured to *not* fsync() on all Append() calls I0627 19:31:43.676317 22252 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:43.626035 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:54712 0627 19:31:43.628685 (+ 2650us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:43.628715 (+ 30us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:43.629876 (+ 1161us) server_negotiation.cc:373] Connection header received 0627 19:31:43.645547 (+ 15671us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:43.645598 (+ 51us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:43.645900 (+ 302us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:43.648731 (+ 2831us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:43.649125 (+ 394us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:43.649744 (+ 619us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:43.651366 (+ 1622us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:43.653484 (+ 2118us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:43.656486 (+ 3002us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:43.658557 (+ 2071us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:43.658598 (+ 41us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:43.658630 (+ 32us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:43.658724 (+ 94us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:43.666345 (+ 7621us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:43.668224 (+ 1879us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:43.668255 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:43.668274 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:43.669031 (+ 757us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:43.669663 (+ 632us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:43.669688 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:43.669706 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:43.670325 (+ 619us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:43.672562 (+ 2237us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:43.672971 (+ 409us) server_negotiation.cc:264] Negotiation successful 0627 19:31:43.673503 (+ 532us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":2383,"thread_start_us":2972,"threads_started":1} I0627 19:31:43.678812 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:31:43.684334 22250 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d: No bootstrap required, opened a new log I0627 19:31:43.708045 22250 raft_consensus.cc:325] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de302f094f4c39abb61c2df94cb27d" member_type: VOTER } I0627 19:31:43.708818 22250 raft_consensus.cc:351] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:43.709061 22250 raft_consensus.cc:628] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7de302f094f4c39abb61c2df94cb27d, State: Initialized, Role: FOLLOWER I0627 19:31:43.709867 22250 consensus_queue.cc:227] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de302f094f4c39abb61c2df94cb27d" member_type: VOTER } I0627 19:31:43.710521 22250 raft_consensus.cc:365] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:43.710784 22250 raft_consensus.cc:457] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:43.711051 22250 raft_consensus.cc:2730] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d: Snoozing failure detection for 1.521s (starting election) I0627 19:31:43.711319 22250 raft_consensus.cc:2782] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:43.718721 22250 raft_consensus.cc:479] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de302f094f4c39abb61c2df94cb27d" member_type: VOTER } I0627 19:31:43.719828 22250 leader_election.cc:270] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:43.726678 22263 raft_consensus.cc:2730] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d: Snoozing failure detection for 2.079s (election complete) I0627 19:31:43.731004 22263 raft_consensus.cc:2537] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:43.741783 22263 raft_consensus.cc:593] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [term 1 LEADER]: Becoming Leader. State: Replica: d7de302f094f4c39abb61c2df94cb27d, State: Running, Role: LEADER I0627 19:31:43.743698 22263 consensus_queue.cc:204] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de302f094f4c39abb61c2df94cb27d" member_type: VOTER } I0627 19:31:43.753651 22252 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:43.719275 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:44788 0627 19:31:43.719574 (+ 299us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:43.719594 (+ 20us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:43.726801 (+ 7207us) server_negotiation.cc:373] Connection header received 0627 19:31:43.727457 (+ 656us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:43.727482 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:43.727596 (+ 114us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:43.727717 (+ 121us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:43.730112 (+ 2395us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:43.730563 (+ 451us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:43.732024 (+ 1461us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:43.733049 (+ 1025us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:43.733215 (+ 166us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:43.737561 (+ 4346us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:43.737653 (+ 92us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:43.737677 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:43.737759 (+ 82us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:43.744686 (+ 6927us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:43.747252 (+ 2566us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:43.747278 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:43.747295 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:43.747919 (+ 624us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:43.749068 (+ 1149us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:43.749093 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:43.749110 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:43.749563 (+ 453us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:43.749965 (+ 402us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:43.751704 (+ 1739us) server_negotiation.cc:264] Negotiation successful 0627 19:31:43.752049 (+ 345us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":85} I0627 19:31:43.758723 22250 sys_catalog.cc:337] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "d7de302f094f4c39abb61c2df94cb27d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de302f094f4c39abb61c2df94cb27d" member_type: VOTER } } I0627 19:31:43.867635 22250 sys_catalog.cc:340] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [sys.catalog]: This master's current role is: LEADER I0627 19:31:43.759281 22264 sys_catalog.cc:337] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7de302f094f4c39abb61c2df94cb27d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de302f094f4c39abb61c2df94cb27d" member_type: VOTER } } I0627 19:31:43.916134 22264 sys_catalog.cc:340] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [sys.catalog]: This master's current role is: LEADER I0627 19:31:43.759395 22265 sys_catalog.cc:337] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7de302f094f4c39abb61c2df94cb27d. Latest consensus state: current_term: 1 leader_uuid: "d7de302f094f4c39abb61c2df94cb27d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7de302f094f4c39abb61c2df94cb27d" member_type: VOTER } } I0627 19:31:43.919924 22265 sys_catalog.cc:340] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [sys.catalog]: This master's current role is: LEADER I0627 19:31:43.921222 22250 sys_catalog.cc:424] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:43.921531 22274 catalog_manager.cc:1006] Loading table and tablet metadata into memory... W0627 19:31:43.940841 22206 tablet.cc:1678] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:43.943183 22181 master_main.cc:86] Master server successfully started. I0627 19:31:43.944459 22275 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0627 19:31:43.954197 22274 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0627 19:31:44.020495 22274 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:44.022711 22274 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:44.055865 22274 catalog_manager.cc:4102] T 00000000000000000000000000000000 P d7de302f094f4c39abb61c2df94cb27d: Generated new TSK 0 I0627 19:31:44.078850 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 22181 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1969 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:44 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22333](info): setting up network... Jun 27 19:31:44 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22333](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:44 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22333](info): skipping unrecognized local address family 17 Jun 27 19:31:44 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22333](info): skipping unrecognized local address family 17 Jun 27 19:31:44 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22333](info): set up 1 sockets Jun 27 19:31:44 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22333](info): commencing operation krb5kdc: starting... W0627 19:31:44.307062 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.182s user 0.004s sys 0.014s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:44 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:31:44.588321 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:45974 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=required --rpc_encrypt_loopback_connections=true --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:44.873064 22359 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:44.873453 22359 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:31:44.873844 22359 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:44.891804 22359 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:44.892074 22359 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:44.892261 22359 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:44.892467 22359 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:44.892673 22359 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:44.892886 22359 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:31:44.939283 22359 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_encrypt_loopback_connections=true --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:45974 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=disabled --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:44.951642 22359 master_main.cc:80] Initializing master server... I0627 19:31:44.955230 22359 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 221092us Jun 27 19:31:44 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153104, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:44.961987 22359 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:31:44.978225 22359 fs_manager.cc:261] Metadata directory not provided I0627 19:31:44.978500 22359 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:44.978891 22359 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:44.979077 22359 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:44.995373 22359 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "aa570ea2204943059365f845738c41f8" format_stamp: "Formatted at 2018-06-28 02:31:44 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:44.996465 22359 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "aa570ea2204943059365f845738c41f8" format_stamp: "Formatted at 2018-06-28 02:31:44 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:45.055456 22359 fs_manager.cc:501] Time spent creating directory manager: real 0.058s user 0.010s sys 0.022s I0627 19:31:45.056713 22359 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:45.057284 22359 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:45.072126 22359 fs_manager.cc:417] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s I0627 19:31:45.072350 22359 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/wal uuid: "aa570ea2204943059365f845738c41f8" format_stamp: "Formatted at 2018-06-28 02:31:44 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:45.072757 22359 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:45.332815 22359 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:45.339958 22359 master_main.cc:83] Starting Master server... I0627 19:31:45.583595 22359 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:45974 I0627 19:31:45.586565 22359 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:31:45.586745 22359 webserver.cc:186] Document root disabled I0627 19:31:45.598242 22359 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:34302/ I0627 19:31:45.599287 22359 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:45.605880 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 22359 I0627 19:31:45.622107 22551 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:45.660966 22553 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:45.608640 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:42256 0627 19:31:45.621380 (+ 12740us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:45.621417 (+ 37us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:45.621485 (+ 68us) server_negotiation.cc:373] Connection header received 0627 19:31:45.649531 (+ 28046us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:45.649585 (+ 54us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:45.649850 (+ 265us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:45.650201 (+ 351us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:45.651220 (+ 1019us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:45.651900 (+ 680us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:45.653457 (+ 1557us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:45.655217 (+ 1760us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:45.655459 (+ 242us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:45.656602 (+ 1143us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:45.656693 (+ 91us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:45.656729 (+ 36us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:45.657347 (+ 618us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:45.657520 (+ 173us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:45.657606 (+ 86us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:45.657911 (+ 305us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:45.658963 (+ 1052us) server_negotiation.cc:264] Negotiation successful 0627 19:31:45.659103 (+ 140us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":12368,"thread_start_us":12107,"threads_started":1} I0627 19:31:45.663942 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:31:45.669812 22551 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8: Bootstrap starting. I0627 19:31:45.674798 22551 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8: No blocks or log segments found. Creating new log. I0627 19:31:45.681828 22551 log.cc:525] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8: Log is configured to *not* fsync() on all Append() calls I0627 19:31:45.705804 22551 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8: No bootstrap required, opened a new log I0627 19:31:45.719265 22553 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:45.699636 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:45351 0627 19:31:45.700059 (+ 423us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:45.700242 (+ 183us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:45.703618 (+ 3376us) server_negotiation.cc:373] Connection header received 0627 19:31:45.703745 (+ 127us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:45.703772 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:45.703896 (+ 124us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:45.704036 (+ 140us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:45.706707 (+ 2671us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:45.707210 (+ 503us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:45.708706 (+ 1496us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:45.709817 (+ 1111us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:45.710008 (+ 191us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:45.711172 (+ 1164us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:45.711236 (+ 64us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:45.711263 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:45.711316 (+ 53us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:45.711392 (+ 76us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:45.711486 (+ 94us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:45.711772 (+ 286us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:45.717686 (+ 5914us) server_negotiation.cc:264] Negotiation successful 0627 19:31:45.717791 (+ 105us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":179} I0627 19:31:45.733044 22551 raft_consensus.cc:325] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa570ea2204943059365f845738c41f8" member_type: VOTER } I0627 19:31:45.896934 22551 raft_consensus.cc:351] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:45.897158 22551 raft_consensus.cc:628] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa570ea2204943059365f845738c41f8, State: Initialized, Role: FOLLOWER I0627 19:31:45.897963 22551 consensus_queue.cc:227] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa570ea2204943059365f845738c41f8" member_type: VOTER } I0627 19:31:45.898485 22551 raft_consensus.cc:365] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:45.898722 22551 raft_consensus.cc:457] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:45.898960 22551 raft_consensus.cc:2730] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8: Snoozing failure detection for 1.560s (starting election) I0627 19:31:45.899205 22551 raft_consensus.cc:2782] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:45.903595 22551 raft_consensus.cc:479] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa570ea2204943059365f845738c41f8" member_type: VOTER } I0627 19:31:45.904283 22551 leader_election.cc:270] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:45.910866 22571 raft_consensus.cc:2730] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8: Snoozing failure detection for 2.197s (election complete) I0627 19:31:45.912139 22571 raft_consensus.cc:2537] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:45.923205 22571 raft_consensus.cc:593] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [term 1 LEADER]: Becoming Leader. State: Replica: aa570ea2204943059365f845738c41f8, State: Running, Role: LEADER I0627 19:31:45.923838 22571 consensus_queue.cc:204] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa570ea2204943059365f845738c41f8" member_type: VOTER } I0627 19:31:45.934152 22573 sys_catalog.cc:337] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa570ea2204943059365f845738c41f8. Latest consensus state: current_term: 1 leader_uuid: "aa570ea2204943059365f845738c41f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa570ea2204943059365f845738c41f8" member_type: VOTER } } I0627 19:31:45.934638 22573 sys_catalog.cc:340] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [sys.catalog]: This master's current role is: LEADER I0627 19:31:45.935564 22551 sys_catalog.cc:337] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "aa570ea2204943059365f845738c41f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa570ea2204943059365f845738c41f8" member_type: VOTER } } I0627 19:31:45.935969 22551 sys_catalog.cc:340] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [sys.catalog]: This master's current role is: LEADER I0627 19:31:45.937263 22551 sys_catalog.cc:424] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:45.934039 22572 sys_catalog.cc:337] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa570ea2204943059365f845738c41f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa570ea2204943059365f845738c41f8" member_type: VOTER } } I0627 19:31:45.938773 22572 sys_catalog.cc:340] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8 [sys.catalog]: This master's current role is: LEADER I0627 19:31:45.945655 22359 master_main.cc:86] Master server successfully started. W0627 19:31:45.958941 22576 catalog_manager.cc:1064] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:45.959224 22576 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:31:45.959501 22577 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:45.962580 22577 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... W0627 19:31:46.099663 22453 tablet.cc:1678] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:46.139073 22577 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:46.143141 22577 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:46.171530 22577 catalog_manager.cc:4102] T 00000000000000000000000000000000 P aa570ea2204943059365f845738c41f8: Generated new TSK 0 I0627 19:31:46.209543 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 22359 W0627 19:31:46.294108 22308 connection.cc:512] client connection to 127.12.131.62:45974 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.62:45974): Connection reset by peer (error 104) [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2185 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:46 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22584](info): setting up network... Jun 27 19:31:46 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22584](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:46 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22584](info): skipping unrecognized local address family 17 Jun 27 19:31:46 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22584](info): skipping unrecognized local address family 17 Jun 27 19:31:46 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22584](info): set up 1 sockets Jun 27 19:31:46 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22584](info): commencing operation krb5kdc: starting... W0627 19:31:46.545424 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.211s user 0.003s sys 0.015s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:46 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:31:46.805276 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:57967 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=required --rpc_encrypt_loopback_connections=false --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:47.148175 22608 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:47.148712 22608 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:31:47.149159 22608 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:47.175704 22608 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:47.176059 22608 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:47.176265 22608 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:47.176486 22608 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:47.176715 22608 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:47.176956 22608 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:31:47.245251 22608 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:57967 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=disabled --rpc_encryption=required --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:47.270264 22608 master_main.cc:80] Initializing master server... I0627 19:31:47.279825 22608 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 222592us Jun 27 19:31:47 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153107, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:47.289392 22608 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:31:47.311044 22608 fs_manager.cc:261] Metadata directory not provided I0627 19:31:47.311288 22608 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:47.313463 22608 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:47.313673 22608 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:47.332847 22608 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "313e653c6dc14c14bc04b55f8cd2aea0" format_stamp: "Formatted at 2018-06-28 02:31:47 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:47.334111 22608 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "313e653c6dc14c14bc04b55f8cd2aea0" format_stamp: "Formatted at 2018-06-28 02:31:47 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:47.362332 22608 fs_manager.cc:501] Time spent creating directory manager: real 0.027s user 0.013s sys 0.006s I0627 19:31:47.364317 22608 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:47.365015 22608 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:47.386132 22608 fs_manager.cc:417] Time spent opening block manager: real 0.010s user 0.003s sys 0.003s I0627 19:31:47.386368 22608 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/wal uuid: "313e653c6dc14c14bc04b55f8cd2aea0" format_stamp: "Formatted at 2018-06-28 02:31:47 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:47.386776 22608 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:47.641471 22608 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:47.645536 22608 master_main.cc:83] Starting Master server... I0627 19:31:47.876451 22608 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:57967 I0627 19:31:47.878127 22608 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:31:47.878311 22608 webserver.cc:186] Document root disabled I0627 19:31:47.880336 22608 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:33669/ I0627 19:31:47.881382 22608 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:47.885874 22732 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:47.885947 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 22608 I0627 19:31:47.921577 22732 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0: Bootstrap starting. I0627 19:31:47.924358 22734 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:47.888728 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:51583 0627 19:31:47.894706 (+ 5978us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:47.894744 (+ 38us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:47.894818 (+ 74us) server_negotiation.cc:373] Connection header received 0627 19:31:47.905281 (+ 10463us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:47.905333 (+ 52us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:47.908219 (+ 2886us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:47.909810 (+ 1591us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:47.910256 (+ 446us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:47.911010 (+ 754us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:47.912774 (+ 1764us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:47.914729 (+ 1955us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:47.916020 (+ 1291us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:47.916664 (+ 644us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:47.916697 (+ 33us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:47.916726 (+ 29us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:47.917542 (+ 816us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:47.917637 (+ 95us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:47.917711 (+ 74us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:47.920915 (+ 3204us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:47.921862 (+ 947us) server_negotiation.cc:264] Negotiation successful 0627 19:31:47.922131 (+ 269us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5630,"thread_start_us":4977,"threads_started":1} I0627 19:31:47.927026 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:31:47.929033 22732 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0: No blocks or log segments found. Creating new log. I0627 19:31:47.930872 22732 log.cc:525] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0: Log is configured to *not* fsync() on all Append() calls I0627 19:31:47.941048 22732 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0: No bootstrap required, opened a new log I0627 19:31:47.963838 22732 raft_consensus.cc:325] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e653c6dc14c14bc04b55f8cd2aea0" member_type: VOTER } I0627 19:31:47.964435 22732 raft_consensus.cc:351] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:47.964676 22732 raft_consensus.cc:628] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313e653c6dc14c14bc04b55f8cd2aea0, State: Initialized, Role: FOLLOWER I0627 19:31:47.965940 22732 consensus_queue.cc:227] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e653c6dc14c14bc04b55f8cd2aea0" member_type: VOTER } I0627 19:31:47.966507 22732 raft_consensus.cc:365] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:47.966768 22732 raft_consensus.cc:457] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:47.967030 22732 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0: Snoozing failure detection for 2.098s (starting election) I0627 19:31:47.967284 22732 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:47.974431 22732 raft_consensus.cc:479] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e653c6dc14c14bc04b55f8cd2aea0" member_type: VOTER } I0627 19:31:47.975183 22732 leader_election.cc:270] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:47.986276 22741 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0: Snoozing failure detection for 2.110s (election complete) I0627 19:31:47.987649 22741 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:47.992131 22734 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:47.970230 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:48921 0627 19:31:47.970586 (+ 356us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:47.970609 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:47.976850 (+ 6241us) server_negotiation.cc:373] Connection header received 0627 19:31:47.977173 (+ 323us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:47.977199 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:47.977321 (+ 122us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:47.977467 (+ 146us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:47.978373 (+ 906us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:47.978778 (+ 405us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:47.980198 (+ 1420us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:47.981242 (+ 1044us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:47.981431 (+ 189us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:47.989212 (+ 7781us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:47.989278 (+ 66us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:47.989303 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:47.989353 (+ 50us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:47.989432 (+ 79us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:47.989519 (+ 87us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:47.989787 (+ 268us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:47.990693 (+ 906us) server_negotiation.cc:264] Negotiation successful 0627 19:31:47.990788 (+ 95us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":154} I0627 19:31:48.002221 22741 raft_consensus.cc:593] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [term 1 LEADER]: Becoming Leader. State: Replica: 313e653c6dc14c14bc04b55f8cd2aea0, State: Running, Role: LEADER I0627 19:31:48.176581 22741 consensus_queue.cc:204] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e653c6dc14c14bc04b55f8cd2aea0" member_type: VOTER } I0627 19:31:48.180277 22732 sys_catalog.cc:337] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "313e653c6dc14c14bc04b55f8cd2aea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e653c6dc14c14bc04b55f8cd2aea0" member_type: VOTER } } I0627 19:31:48.180799 22732 sys_catalog.cc:340] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [sys.catalog]: This master's current role is: LEADER I0627 19:31:48.182097 22743 sys_catalog.cc:337] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "313e653c6dc14c14bc04b55f8cd2aea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e653c6dc14c14bc04b55f8cd2aea0" member_type: VOTER } } I0627 19:31:48.182559 22743 sys_catalog.cc:340] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [sys.catalog]: This master's current role is: LEADER I0627 19:31:48.183035 22742 sys_catalog.cc:337] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 313e653c6dc14c14bc04b55f8cd2aea0. Latest consensus state: current_term: 1 leader_uuid: "313e653c6dc14c14bc04b55f8cd2aea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313e653c6dc14c14bc04b55f8cd2aea0" member_type: VOTER } } I0627 19:31:48.183506 22742 sys_catalog.cc:340] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [sys.catalog]: This master's current role is: LEADER I0627 19:31:48.192987 22732 sys_catalog.cc:424] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:48.195479 22745 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:48.198101 22608 master_main.cc:86] Master server successfully started. I0627 19:31:48.202582 22745 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0627 19:31:48.243860 22745 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:48.246909 22745 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:48.264276 22745 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 313e653c6dc14c14bc04b55f8cd2aea0: Generated new TSK 0 I0627 19:31:48.274459 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 22608 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2002 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:48 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22752](info): setting up network... Jun 27 19:31:48 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22752](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:48 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22752](info): skipping unrecognized local address family 17 Jun 27 19:31:48 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22752](info): skipping unrecognized local address family 17 Jun 27 19:31:48 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22752](info): set up 1 sockets Jun 27 19:31:48 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22752](info): commencing operation krb5kdc: starting... W0627 19:31:48.450721 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.138s user 0.004s sys 0.011s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:48 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:31:48.786345 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:55184 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=disabled --rpc_encrypt_loopback_connections=true --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:49.029302 22801 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:49.029884 22801 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:31:49.030931 22801 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:49.073114 22801 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:49.077536 22801 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:49.077724 22801 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:49.077891 22801 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:49.078079 22801 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:49.078310 22801 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:31:49.182240 22801 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_encrypt_loopback_connections=true --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:55184 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=disabled --rpc_encryption=disabled --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:49.202433 22801 master_main.cc:80] Initializing master server... I0627 19:31:49.208724 22801 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 223592us Jun 27 19:31:49 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153109, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:49.218142 22801 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:31:49.235025 22801 fs_manager.cc:261] Metadata directory not provided I0627 19:31:49.235280 22801 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:49.235733 22801 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:49.235925 22801 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:49.252233 22801 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "d9b823f723bf45129c136aa33835d76a" format_stamp: "Formatted at 2018-06-28 02:31:49 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:49.253628 22801 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "d9b823f723bf45129c136aa33835d76a" format_stamp: "Formatted at 2018-06-28 02:31:49 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:49.279726 22801 fs_manager.cc:501] Time spent creating directory manager: real 0.025s user 0.013s sys 0.007s I0627 19:31:49.281682 22801 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:49.283157 22801 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:49.306529 22801 fs_manager.cc:417] Time spent opening block manager: real 0.012s user 0.004s sys 0.008s I0627 19:31:49.306782 22801 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/wal uuid: "d9b823f723bf45129c136aa33835d76a" format_stamp: "Formatted at 2018-06-28 02:31:49 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:49.307152 22801 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:49.770148 22801 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:49.791589 22801 master_main.cc:83] Starting Master server... I0627 19:31:50.018733 22801 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:55184 I0627 19:31:50.019872 22801 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:31:50.019990 22801 webserver.cc:186] Document root disabled I0627 19:31:50.022402 22801 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:47740/ I0627 19:31:50.023320 22801 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:50.029222 22867 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:50.031448 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 22801 I0627 19:31:50.061637 22869 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:50.033743 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:37102 0627 19:31:50.039302 (+ 5559us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:50.039342 (+ 40us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:50.039437 (+ 95us) server_negotiation.cc:373] Connection header received 0627 19:31:50.052606 (+ 13169us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:50.052660 (+ 54us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:50.053140 (+ 480us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:50.055040 (+ 1900us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:50.057827 (+ 2787us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:50.057868 (+ 41us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:50.057900 (+ 32us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:50.057953 (+ 53us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:50.058135 (+ 182us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:50.058251 (+ 116us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:50.059078 (+ 827us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:50.059391 (+ 313us) server_negotiation.cc:264] Negotiation successful 0627 19:31:50.059540 (+ 149us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":3880,"thread_start_us":3293,"threads_started":1} I0627 19:31:50.065832 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:31:50.069608 22867 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a: Bootstrap starting. I0627 19:31:50.072960 22867 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a: No blocks or log segments found. Creating new log. I0627 19:31:50.074983 22867 log.cc:525] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a: Log is configured to *not* fsync() on all Append() calls I0627 19:31:50.086392 22867 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a: No bootstrap required, opened a new log I0627 19:31:50.090363 22869 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:50.084430 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:53429 0627 19:31:50.084794 (+ 364us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:50.084815 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:50.086340 (+ 1525us) server_negotiation.cc:373] Connection header received 0627 19:31:50.086734 (+ 394us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:50.086760 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:50.086882 (+ 122us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:50.087029 (+ 147us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:50.087834 (+ 805us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:50.087860 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:50.087881 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:50.087924 (+ 43us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:50.088017 (+ 93us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:50.088090 (+ 73us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:50.088200 (+ 110us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:50.088672 (+ 472us) server_negotiation.cc:264] Negotiation successful 0627 19:31:50.088765 (+ 93us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":148} I0627 19:31:50.102658 22867 raft_consensus.cc:325] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b823f723bf45129c136aa33835d76a" member_type: VOTER } I0627 19:31:50.179253 22867 raft_consensus.cc:351] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:50.179525 22867 raft_consensus.cc:628] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b823f723bf45129c136aa33835d76a, State: Initialized, Role: FOLLOWER I0627 19:31:50.180229 22867 consensus_queue.cc:227] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b823f723bf45129c136aa33835d76a" member_type: VOTER } I0627 19:31:50.180631 22867 raft_consensus.cc:365] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:50.180805 22867 raft_consensus.cc:457] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:50.180977 22867 raft_consensus.cc:2730] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a: Snoozing failure detection for 1.964s (starting election) I0627 19:31:50.181145 22867 raft_consensus.cc:2782] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:50.183820 22867 raft_consensus.cc:479] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b823f723bf45129c136aa33835d76a" member_type: VOTER } I0627 19:31:50.184322 22867 leader_election.cc:270] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:50.188896 22876 raft_consensus.cc:2730] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a: Snoozing failure detection for 1.503s (election complete) I0627 19:31:50.189241 22876 raft_consensus.cc:2537] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:50.200353 22876 raft_consensus.cc:593] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [term 1 LEADER]: Becoming Leader. State: Replica: d9b823f723bf45129c136aa33835d76a, State: Running, Role: LEADER I0627 19:31:50.201086 22876 consensus_queue.cc:204] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b823f723bf45129c136aa33835d76a" member_type: VOTER } I0627 19:31:50.205555 22867 sys_catalog.cc:337] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "d9b823f723bf45129c136aa33835d76a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b823f723bf45129c136aa33835d76a" member_type: VOTER } } I0627 19:31:50.206184 22867 sys_catalog.cc:340] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [sys.catalog]: This master's current role is: LEADER I0627 19:31:50.208878 22877 sys_catalog.cc:337] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9b823f723bf45129c136aa33835d76a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b823f723bf45129c136aa33835d76a" member_type: VOTER } } I0627 19:31:50.209473 22877 sys_catalog.cc:340] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [sys.catalog]: This master's current role is: LEADER I0627 19:31:50.209317 22878 sys_catalog.cc:337] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b823f723bf45129c136aa33835d76a. Latest consensus state: current_term: 1 leader_uuid: "d9b823f723bf45129c136aa33835d76a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9b823f723bf45129c136aa33835d76a" member_type: VOTER } } I0627 19:31:50.209841 22878 sys_catalog.cc:340] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [sys.catalog]: This master's current role is: LEADER I0627 19:31:50.213343 22867 sys_catalog.cc:424] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:50.213886 22880 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:50.219435 22801 master_main.cc:86] Master server successfully started. I0627 19:31:50.224997 22880 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0627 19:31:50.265084 22880 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:50.265609 22880 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:50.285300 22880 catalog_manager.cc:4102] T 00000000000000000000000000000000 P d9b823f723bf45129c136aa33835d76a: Generated new TSK 0 I0627 19:31:50.309398 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 22801 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2033 ms) [ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:50 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22887](info): setting up network... Jun 27 19:31:50 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22887](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:50 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22887](info): skipping unrecognized local address family 17 Jun 27 19:31:50 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22887](info): skipping unrecognized local address family 17 Jun 27 19:31:50 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22887](info): set up 1 sockets Jun 27 19:31:50 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22887](info): commencing operation krb5kdc: starting... W0627 19:31:50.465492 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.118s user 0.001s sys 0.016s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:50 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:31:50.626574 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:57940 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --user-acl=* --rpc-authentication=disabled --rpc-encryption=disabled --rpc_encrypt_loopback_connections=false --rpc_reuseport=true with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:50.807621 22896 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:50.807963 22896 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:31:50.808218 22896 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:50.820366 22896 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:50.820559 22896 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:50.820674 22896 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:50.820782 22896 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:50.820905 22896 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:50.821036 22896 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:31:50.848037 22896 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:57940 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=disabled --rpc_encryption=disabled --superuser_acl=test-admin --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:50.856508 22896 master_main.cc:80] Initializing master server... I0627 19:31:50.859354 22896 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 224092us Jun 27 19:31:50 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[22887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153110, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:50.864691 22896 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:31:50.872424 22896 fs_manager.cc:261] Metadata directory not provided I0627 19:31:50.872578 22896 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:50.872860 22896 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:50.872983 22896 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:50.882182 22896 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "d158c7a18edf4f28a9ae154134c11850" format_stamp: "Formatted at 2018-06-28 02:31:50 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:50.882930 22896 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "d158c7a18edf4f28a9ae154134c11850" format_stamp: "Formatted at 2018-06-28 02:31:50 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:50.901386 22896 fs_manager.cc:501] Time spent creating directory manager: real 0.018s user 0.011s sys 0.006s I0627 19:31:50.903151 22896 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:50.904625 22896 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:50.922448 22896 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.002s sys 0.004s I0627 19:31:50.922698 22896 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/wal uuid: "d158c7a18edf4f28a9ae154134c11850" format_stamp: "Formatted at 2018-06-28 02:31:50 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:50.923084 22896 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:51.134392 22896 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:51.147806 22896 master_main.cc:83] Starting Master server... I0627 19:31:51.272774 22896 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:57940 I0627 19:31:51.273640 22896 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:31:51.273746 22896 webserver.cc:186] Document root disabled I0627 19:31:51.276048 22896 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:49549/ I0627 19:31:51.277088 22896 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:51.279925 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 22896 I0627 19:31:51.281960 22998 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:51.307214 23000 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:51.282281 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:42512 0627 19:31:51.295775 (+ 13494us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:51.295815 (+ 40us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:51.295898 (+ 83us) server_negotiation.cc:373] Connection header received 0627 19:31:51.302236 (+ 6338us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:51.302291 (+ 55us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:51.302590 (+ 299us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:51.302949 (+ 359us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:51.303829 (+ 880us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:51.303862 (+ 33us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:51.303889 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:51.303938 (+ 49us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:51.304029 (+ 91us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:51.304102 (+ 73us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:51.305016 (+ 914us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:51.305234 (+ 218us) server_negotiation.cc:264] Negotiation successful 0627 19:31:51.305404 (+ 170us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":13121,"thread_start_us":12472,"threads_started":1} I0627 19:31:51.309782 12812 external_mini_cluster.cc:490] 0 TS(s) registered with all masters I0627 19:31:51.314251 22998 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850: Bootstrap starting. I0627 19:31:51.316392 22998 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850: No blocks or log segments found. Creating new log. I0627 19:31:51.318181 22998 log.cc:525] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850: Log is configured to *not* fsync() on all Append() calls I0627 19:31:51.322912 22998 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850: No bootstrap required, opened a new log I0627 19:31:51.333206 22998 raft_consensus.cc:325] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158c7a18edf4f28a9ae154134c11850" member_type: VOTER } I0627 19:31:51.333576 22998 raft_consensus.cc:351] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0627 19:31:51.333740 22998 raft_consensus.cc:628] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d158c7a18edf4f28a9ae154134c11850, State: Initialized, Role: FOLLOWER I0627 19:31:51.334254 22998 consensus_queue.cc:227] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158c7a18edf4f28a9ae154134c11850" member_type: VOTER } I0627 19:31:51.334682 22998 raft_consensus.cc:365] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0627 19:31:51.334852 22998 raft_consensus.cc:457] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0627 19:31:51.335021 22998 raft_consensus.cc:2730] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850: Snoozing failure detection for 2.168s (starting election) I0627 19:31:51.335189 22998 raft_consensus.cc:2782] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:31:51.337640 22998 raft_consensus.cc:479] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158c7a18edf4f28a9ae154134c11850" member_type: VOTER } I0627 19:31:51.338071 22998 leader_election.cc:270] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:31:51.341362 23006 raft_consensus.cc:2730] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850: Snoozing failure detection for 1.985s (election complete) I0627 19:31:51.341778 23006 raft_consensus.cc:2537] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:31:51.349061 23000 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:51.338987 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.0.0.1:42594 0627 19:31:51.339326 (+ 339us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:51.339348 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:51.344937 (+ 5589us) server_negotiation.cc:373] Connection header received 0627 19:31:51.345511 (+ 574us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:51.345541 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:51.345647 (+ 106us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:51.345770 (+ 123us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:51.347113 (+ 1343us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:51.347141 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:51.347169 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: PLAIN 0627 19:31:51.347221 (+ 52us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:51.347300 (+ 79us) server_negotiation.cc:956] Received PLAIN auth, user=jenkins 0627 19:31:51.347361 (+ 61us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:51.347496 (+ 135us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:51.348149 (+ 653us) server_negotiation.cc:264] Negotiation successful 0627 19:31:51.348239 (+ 90us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":117} I0627 19:31:51.354547 23006 raft_consensus.cc:593] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [term 1 LEADER]: Becoming Leader. State: Replica: d158c7a18edf4f28a9ae154134c11850, State: Running, Role: LEADER I0627 19:31:51.509486 23006 consensus_queue.cc:204] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158c7a18edf4f28a9ae154134c11850" member_type: VOTER } I0627 19:31:51.522253 22998 sys_catalog.cc:337] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "d158c7a18edf4f28a9ae154134c11850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158c7a18edf4f28a9ae154134c11850" member_type: VOTER } } I0627 19:31:51.522799 22998 sys_catalog.cc:340] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [sys.catalog]: This master's current role is: LEADER I0627 19:31:51.523033 23008 sys_catalog.cc:337] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d158c7a18edf4f28a9ae154134c11850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158c7a18edf4f28a9ae154134c11850" member_type: VOTER } } I0627 19:31:51.524695 23008 sys_catalog.cc:340] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [sys.catalog]: This master's current role is: LEADER I0627 19:31:51.523824 23009 sys_catalog.cc:337] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d158c7a18edf4f28a9ae154134c11850. Latest consensus state: current_term: 1 leader_uuid: "d158c7a18edf4f28a9ae154134c11850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d158c7a18edf4f28a9ae154134c11850" member_type: VOTER } } I0627 19:31:51.525737 23009 sys_catalog.cc:340] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [sys.catalog]: This master's current role is: LEADER I0627 19:31:51.528699 22998 sys_catalog.cc:424] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:51.529522 23031 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:31:51.533857 22896 master_main.cc:86] Master server successfully started. I0627 19:31:51.544468 23031 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0627 19:31:51.634104 23031 catalog_manager.cc:904] Generated new certificate authority record I0627 19:31:51.634611 23031 catalog_manager.cc:1026] Loading token signing keys... I0627 19:31:51.651372 23031 catalog_manager.cc:4102] T 00000000000000000000000000000000 P d158c7a18edf4f28a9ae154134c11850: Generated new TSK 0 I0627 19:31:51.669669 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 22896 [ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1358 ms) [----------] 12 tests from AuthTokenIssuingTest (24801 ms total) [----------] 2 tests from ConnectToFollowerMasterTest [ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:31:51 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): setting up network... Jun 27 19:31:51 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:31:51 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): skipping unrecognized local address family 17 Jun 27 19:31:51 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): skipping unrecognized local address family 17 Jun 27 19:31:51 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): set up 1 sockets Jun 27 19:31:51 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): commencing operation krb5kdc: starting... W0627 19:31:51.891813 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.186s user 0.003s sys 0.014s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:31:52 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. I0627 19:31:52.152794 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:51131 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --enable_leader_failure_detection=false --rpc_authentication=required --rpc_encryption=optional --rpc_reuseport=true --master_addresses=127.12.131.62:51131,127.12.131.61:46323,127.12.131.60:44480 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:52.416081 23061 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0627 19:31:52.416813 23061 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:52.417019 23061 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:31:52.417417 23061 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:52.435640 23061 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:52.435923 23061 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:52.436131 23061 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:52.436314 23061 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:52.436533 23061 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:52.436750 23061 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:31:52.476549 23061 master_main.cc:73] Master server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --master_addresses=127.12.131.62:51131,127.12.131.61:46323,127.12.131.60:44480 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:51131 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:52.490891 23061 master_main.cc:80] Initializing master server... I0627 19:31:52.494272 23061 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 225092us Jun 27 19:31:52 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153112, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:52.500805 23061 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:31:52.510464 23061 fs_manager.cc:261] Metadata directory not provided I0627 19:31:52.510715 23061 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:52.511175 23061 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:52.511368 23061 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:52.526628 23061 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "b9d10270bf864c51bdcf2f696e252fdd" format_stamp: "Formatted at 2018-06-28 02:31:52 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:52.527724 23061 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "b9d10270bf864c51bdcf2f696e252fdd" format_stamp: "Formatted at 2018-06-28 02:31:52 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:52.558189 23061 fs_manager.cc:501] Time spent creating directory manager: real 0.030s user 0.014s sys 0.007s I0627 19:31:52.561146 23061 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:52.562506 23061 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:52.584533 23061 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s I0627 19:31:52.584725 23061 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/wal uuid: "b9d10270bf864c51bdcf2f696e252fdd" format_stamp: "Formatted at 2018-06-28 02:31:52 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:52.585059 23061 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:52.865882 23061 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:52.869164 23061 master_main.cc:83] Starting Master server... I0627 19:31:53.108274 23061 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:51131 I0627 19:31:53.110193 23061 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:31:53.111199 23061 webserver.cc:186] Document root disabled I0627 19:31:53.115525 23061 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:37852/ I0627 19:31:53.116576 23061 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:31:53.121683 23209 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:53.124220 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 23061 I0627 19:31:53.143031 23209 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } has no permanent_uuid. Determining permanent_uuid... WARNING: no policy specified for kudu/127.12.131.61@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.61@KRBTEST.COM" created. Jun 27 19:31:53 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153112, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:53.186028 23214 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:53.146802 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:51131 0627 19:31:53.154099 (+ 7297us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:53.154174 (+ 75us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:53.160157 (+ 5983us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:53.161534 (+ 1377us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:53.161564 (+ 30us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:53.162396 (+ 832us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:53.162638 (+ 242us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:53.162937 (+ 299us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:53.162982 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:53.163888 (+ 906us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:53.163904 (+ 16us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:53.164719 (+ 815us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:53.164757 (+ 38us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:53.167298 (+ 2541us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:53.167324 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:53.167617 (+ 293us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:53.167797 (+ 180us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:53.167819 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:53.170709 (+ 2890us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:53.177752 (+ 7043us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:53.177781 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:53.177805 (+ 24us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:53.179068 (+ 1263us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:53.180301 (+ 1233us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:53.180326 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:53.180346 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:53.180560 (+ 214us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:53.182581 (+ 2021us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:53.182609 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:53.182935 (+ 326us) client_negotiation.cc:695] Sending connection context 0627 19:31:53.183204 (+ 269us) client_negotiation.cc:225] Negotiation successful 0627 19:31:53.183770 (+ 566us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":6963,"thread_start_us":2088,"threads_started":1} I0627 19:31:53.188750 23216 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:53.151713 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:47931 0627 19:31:53.157648 (+ 5935us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:53.157683 (+ 35us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:53.157749 (+ 66us) server_negotiation.cc:373] Connection header received 0627 19:31:53.160601 (+ 2852us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:53.160647 (+ 46us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:53.160860 (+ 213us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:53.161075 (+ 215us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:53.163181 (+ 2106us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:53.163716 (+ 535us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:53.164934 (+ 1218us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:53.166716 (+ 1782us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:53.166906 (+ 190us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:53.170970 (+ 4064us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:53.171004 (+ 34us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:53.171030 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:53.171099 (+ 69us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:53.177525 (+ 6426us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:53.179259 (+ 1734us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:53.179286 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:53.179303 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:53.180135 (+ 832us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:53.180727 (+ 592us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:53.180750 (+ 23us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:53.180767 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:53.181328 (+ 561us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:53.181560 (+ 232us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:53.186665 (+ 5105us) server_negotiation.cc:264] Negotiation successful 0627 19:31:53.187098 (+ 433us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5687,"thread_start_us":6650,"threads_started":1} I0627 19:31:53.191262 23209 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } has no permanent_uuid. Determining permanent_uuid... I0627 19:31:53.196105 23214 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:53.192518 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:46323 0627 19:31:53.195515 (+ 2997us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:53.195683 (+ 168us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.61:46323: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":2777} W0627 19:31:53.196957 23209 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.61:46323: Network error: Client connection negotiation failed: client connection to 127.12.131.61:46323: connect: Connection refused (error 111) Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.61.keytab. Entry for principal kudu/127.12.131.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.61.keytab. Entry for principal kudu/127.12.131.61 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.61.keytab. Entry for principal kudu/127.12.131.61 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.61.keytab. I0627 19:31:53.204063 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.61:46323 --webserver_interface=127.12.131.61 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.61.keytab --principal=kudu/127.12.131.61 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --enable_leader_failure_detection=false --rpc_authentication=required --rpc_encryption=optional --rpc_reuseport=true --master_addresses=127.12.131.62:51131,127.12.131.61:46323,127.12.131.60:44480 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:31:53.217566 23209 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } attempt: 1 I0627 19:31:53.219244 23214 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:53.218326 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:46323 0627 19:31:53.218645 (+ 319us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:53.218800 (+ 155us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.61:46323: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":111} W0627 19:31:53.219637 23209 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.61:46323: Network error: Client connection negotiation failed: client connection to 127.12.131.61:46323: connect: Connection refused (error 111) I0627 19:31:53.269558 23209 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } attempt: 2 I0627 19:31:53.271224 23214 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:53.270300 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:46323 0627 19:31:53.270641 (+ 341us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:53.270790 (+ 149us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.61:46323: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":130} W0627 19:31:53.271575 23209 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.61:46323: Network error: Client connection negotiation failed: client connection to 127.12.131.61:46323: connect: Connection refused (error 111) I0627 19:31:53.382932 23209 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } attempt: 3 I0627 19:31:53.384554 23214 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:53.383689 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:46323 0627 19:31:53.383978 (+ 289us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:53.384134 (+ 156us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.61:46323: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":79} W0627 19:31:53.384882 23209 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.61:46323: Network error: Client connection negotiation failed: client connection to 127.12.131.61:46323: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:53.437893 23223 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0627 19:31:53.438627 23223 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:53.438817 23223 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.61 W0627 19:31:53.439204 23223 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:53.457250 23223 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:53.457584 23223 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:53.457792 23223 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:53.457998 23223 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:53.458226 23223 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:53.458497 23223 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.61 I0627 19:31:53.496059 23223 master_main.cc:73] Master server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/wal --ipki_ca_key_size=1024 --master_addresses=127.12.131.62:51131,127.12.131.61:46323,127.12.131.60:44480 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.61:46323 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.61.keytab --principal=kudu/127.12.131.61 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/data/info.pb --webserver_interface=127.12.131.61 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:53.511343 23223 master_main.cc:80] Initializing master server... I0627 19:31:53.513589 23223 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 225592us Jun 27 19:31:53 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153113, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:53.516741 23223 init.cc:339] Logged in from keytab as kudu/127.12.131.61@KRBTEST.COM (short username kudu) I0627 19:31:53.528684 23223 fs_manager.cc:261] Metadata directory not provided I0627 19:31:53.528923 23223 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:53.529300 23223 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:53.529510 23223 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:53.544968 23223 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/data/instance: uuid: "c7c68ec568384c83885614bc22ec1a50" format_stamp: "Formatted at 2018-06-28 02:31:53 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:53.546042 23223 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/wal/instance: uuid: "c7c68ec568384c83885614bc22ec1a50" format_stamp: "Formatted at 2018-06-28 02:31:53 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:53.561246 23209 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } attempt: 4 I0627 19:31:53.563010 23214 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:53.562020 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:46323 0627 19:31:53.562439 (+ 419us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:53.562605 (+ 166us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.61:46323: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":229} W0627 19:31:53.563436 23209 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.61:46323: Network error: Client connection negotiation failed: client connection to 127.12.131.61:46323: connect: Connection refused (error 111) I0627 19:31:53.574352 23223 fs_manager.cc:501] Time spent creating directory manager: real 0.028s user 0.012s sys 0.006s I0627 19:31:53.576308 23223 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:53.577008 23223 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:53.595084 23223 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.002s sys 0.004s I0627 19:31:53.595343 23223 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/wal uuid: "c7c68ec568384c83885614bc22ec1a50" format_stamp: "Formatted at 2018-06-28 02:31:53 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:53.595751 23223 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:53.867508 23209 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } attempt: 5 I0627 19:31:53.869594 23214 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:53.868300 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:46323 0627 19:31:53.868817 (+ 517us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:53.869003 (+ 186us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.61:46323: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":272} W0627 19:31:53.870179 23209 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.61:46323: Network error: Client connection negotiation failed: client connection to 127.12.131.61:46323: connect: Connection refused (error 111) I0627 19:31:53.928393 23223 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:53.933723 23223 master_main.cc:83] Starting Master server... I0627 19:31:54.169860 23223 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.61:46323 I0627 19:31:54.171627 23223 webserver.cc:175] Starting webserver on 127.12.131.61:0 I0627 19:31:54.171819 23223 webserver.cc:186] Document root disabled I0627 19:31:54.178256 23223 webserver.cc:313] Webserver started. Bound to: http://127.12.131.61:51465/ I0627 19:31:54.179533 23223 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-1/data/info.pb I0627 19:31:54.184780 23393 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:54.187855 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 23223 I0627 19:31:54.214334 23393 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } has no permanent_uuid. Determining permanent_uuid... WARNING: no policy specified for kudu/127.12.131.60@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.60@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.60.keytab. Entry for principal kudu/127.12.131.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.60.keytab. Entry for principal kudu/127.12.131.60 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.60.keytab. Entry for principal kudu/127.12.131.60 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.60.keytab. I0627 19:31:54.249600 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.60:44480 --webserver_interface=127.12.131.60 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.60.keytab --principal=kudu/127.12.131.60 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --enable_leader_failure_detection=false --rpc_authentication=required --rpc_encryption=optional --rpc_reuseport=true --master_addresses=127.12.131.62:51131,127.12.131.61:46323,127.12.131.60:44480 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} Jun 27 19:31:54 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153113, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.61@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:54.289585 23397 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.219624 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:47940 0627 19:31:54.220975 (+ 1351us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:54.221000 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:54.241610 (+ 20610us) server_negotiation.cc:373] Connection header received 0627 19:31:54.249775 (+ 8165us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:54.249803 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:54.249931 (+ 128us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:54.250068 (+ 137us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:54.259644 (+ 9576us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:54.260031 (+ 387us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:54.261494 (+ 1463us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:54.262569 (+ 1075us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:54.262752 (+ 183us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:54.272816 (+ 10064us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:54.272846 (+ 30us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:54.272867 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:54.272945 (+ 78us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:54.279443 (+ 6498us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:54.284512 (+ 5069us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:54.284540 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:54.284558 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:54.285177 (+ 619us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:54.285801 (+ 624us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:54.285829 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:54.285847 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:54.286302 (+ 455us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:54.286514 (+ 212us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:54.287436 (+ 922us) server_negotiation.cc:264] Negotiation successful 0627 19:31:54.287845 (+ 409us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1147,"thread_start_us":905,"threads_started":1} I0627 19:31:54.290163 23398 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.219800 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:51131 0627 19:31:54.241341 (+ 21541us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.241449 (+ 108us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:54.249444 (+ 7995us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:54.257752 (+ 8308us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:54.257789 (+ 37us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:54.258694 (+ 905us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:54.259005 (+ 311us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:54.259338 (+ 333us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:54.259409 (+ 71us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:54.260227 (+ 818us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:54.260252 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:54.261231 (+ 979us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:54.261271 (+ 40us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:54.262811 (+ 1540us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:54.262839 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:54.263138 (+ 299us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:54.263405 (+ 267us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:54.263423 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:54.272464 (+ 9041us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:54.279663 (+ 7199us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:54.279696 (+ 33us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:54.279722 (+ 26us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:54.281059 (+ 1337us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:54.285369 (+ 4310us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:54.285396 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:54.285416 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:54.285640 (+ 224us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:54.286503 (+ 863us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:54.286533 (+ 30us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:54.286942 (+ 409us) client_negotiation.cc:695] Sending connection context 0627 19:31:54.287250 (+ 308us) client_negotiation.cc:225] Negotiation successful 0627 19:31:54.287835 (+ 585us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":21156,"thread_start_us":23472,"threads_started":1} I0627 19:31:54.292711 23393 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } has no permanent_uuid. Determining permanent_uuid... Jun 27 19:31:54 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153113, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.61@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM I0627 19:31:54.337936 23398 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.293676 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:46323 0627 19:31:54.294136 (+ 460us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.294178 (+ 42us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:54.294318 (+ 140us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:54.308685 (+ 14367us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:54.308723 (+ 38us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:54.309447 (+ 724us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:54.309591 (+ 144us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:54.309755 (+ 164us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:54.309794 (+ 39us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:54.310722 (+ 928us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:54.310747 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:54.311527 (+ 780us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:54.311568 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:54.313887 (+ 2319us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:54.313914 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:54.314193 (+ 279us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:54.314350 (+ 157us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:54.314367 (+ 17us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:54.316921 (+ 2554us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:54.325038 (+ 8117us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:54.325065 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:54.325085 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:54.326309 (+ 1224us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:54.327861 (+ 1552us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:54.327884 (+ 23us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:54.327901 (+ 17us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:54.328045 (+ 144us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:54.334818 (+ 6773us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:54.334844 (+ 26us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:54.335141 (+ 297us) client_negotiation.cc:695] Sending connection context 0627 19:31:54.335364 (+ 223us) client_negotiation.cc:225] Negotiation successful 0627 19:31:54.335892 (+ 528us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":5} I0627 19:31:54.341681 23406 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.294388 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60546 0627 19:31:54.307905 (+ 13517us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:54.307936 (+ 31us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:54.307990 (+ 54us) server_negotiation.cc:373] Connection header received 0627 19:31:54.308122 (+ 132us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:54.308149 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:54.308345 (+ 196us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:54.308565 (+ 220us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:54.309957 (+ 1392us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:54.310554 (+ 597us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:54.311749 (+ 1195us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:54.313676 (+ 1927us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:54.313864 (+ 188us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:54.317200 (+ 3336us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:54.317240 (+ 40us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:54.317269 (+ 29us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:54.317344 (+ 75us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:54.324820 (+ 7476us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:54.326584 (+ 1764us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:54.326615 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:54.326635 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:54.327704 (+ 1069us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:54.333917 (+ 6213us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:54.333946 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:54.333965 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:54.334597 (+ 632us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:54.338909 (+ 4312us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:54.339188 (+ 279us) server_negotiation.cc:264] Negotiation successful 0627 19:31:54.339702 (+ 514us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":458,"server-negotiator.queue_time_us":13221,"thread_start_us":4990,"threads_started":1} I0627 19:31:54.344455 23393 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } has no permanent_uuid. Determining permanent_uuid... I0627 19:31:54.346264 23398 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.345289 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:54.345629 (+ 340us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.345822 (+ 193us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":113} W0627 19:31:54.346840 23393 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:44480: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) I0627 19:31:54.366271 23393 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } attempt: 1 I0627 19:31:54.368170 23398 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.367204 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:54.367552 (+ 348us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.367713 (+ 161us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":102} W0627 19:31:54.368650 23393 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:44480: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) I0627 19:31:54.417011 23393 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } attempt: 2 I0627 19:31:54.418597 23398 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.417728 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:54.418005 (+ 277us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.418157 (+ 152us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":71} W0627 19:31:54.419148 23393 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:44480: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) I0627 19:31:54.404582 23209 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } attempt: 6 Jun 27 19:31:54 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153112, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM I0627 19:31:54.513113 23406 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.483449 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60550 0627 19:31:54.483763 (+ 314us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:54.483784 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:54.489484 (+ 5700us) server_negotiation.cc:373] Connection header received 0627 19:31:54.489718 (+ 234us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:54.489736 (+ 18us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:54.489823 (+ 87us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:54.489931 (+ 108us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:54.490949 (+ 1018us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:54.491298 (+ 349us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:54.492721 (+ 1423us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:54.493817 (+ 1096us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:54.493992 (+ 175us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:54.497131 (+ 3139us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:54.497149 (+ 18us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:54.497161 (+ 12us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:54.497213 (+ 52us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:54.500917 (+ 3704us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:54.502624 (+ 1707us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:54.502652 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:54.502671 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:54.503293 (+ 622us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:54.503952 (+ 659us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:54.503981 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:54.504000 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:54.505127 (+ 1127us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:54.505308 (+ 181us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:54.510901 (+ 5593us) server_negotiation.cc:264] Negotiation successful 0627 19:31:54.511347 (+ 446us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":121} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:54.513871 23402 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0627 19:31:54.514678 23402 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:54.514969 23402 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.60 W0627 19:31:54.515527 23402 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0627 19:31:54.517325 23429 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.482508 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:46323 0627 19:31:54.489344 (+ 6836us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.489389 (+ 45us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:54.489555 (+ 166us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:54.490029 (+ 474us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:54.490054 (+ 25us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:54.490537 (+ 483us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:54.490662 (+ 125us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:54.490771 (+ 109us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:54.490798 (+ 27us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:54.491545 (+ 747us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:54.491571 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:54.492430 (+ 859us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:54.492474 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:54.494504 (+ 2030us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:54.494533 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:54.494802 (+ 269us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:54.494946 (+ 144us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:54.494958 (+ 12us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:54.496916 (+ 1958us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:54.501145 (+ 4229us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:54.501171 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:54.501190 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:54.502405 (+ 1215us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:54.503568 (+ 1163us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:54.503594 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:54.503615 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:54.503772 (+ 157us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:54.509457 (+ 5685us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:54.509485 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:54.509821 (+ 336us) client_negotiation.cc:695] Sending connection context 0627 19:31:54.513806 (+ 3985us) client_negotiation.cc:225] Negotiation successful 0627 19:31:54.514436 (+ 630us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":6610,"thread_start_us":6297,"threads_started":1} I0627 19:31:54.519775 23209 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } has no permanent_uuid. Determining permanent_uuid... I0627 19:31:54.521586 23429 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.520594 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:54.521013 (+ 419us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.521157 (+ 144us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":232} W0627 19:31:54.522001 23209 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:44480: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) W0627 19:31:54.533208 23402 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:31:54.533550 23402 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:54.533726 23402 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:54.533900 23402 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:31:54.534103 23402 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:31:54.534328 23402 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.60 I0627 19:31:54.554344 23209 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } attempt: 1 I0627 19:31:54.556330 23429 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.555214 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:54.555708 (+ 494us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.555868 (+ 160us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":282} W0627 19:31:54.556957 23209 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:44480: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) I0627 19:31:54.591619 23402 master_main.cc:73] Master server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/wal --ipki_ca_key_size=1024 --master_addresses=127.12.131.62:51131,127.12.131.61:46323,127.12.131.60:44480 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.60:44480 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.60.keytab --principal=kudu/127.12.131.60 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/data/info.pb --webserver_interface=127.12.131.60 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:54.530474 23393 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } attempt: 3 I0627 19:31:54.611716 23398 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.610851 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:54.611126 (+ 275us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.611271 (+ 145us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":76} W0627 19:31:54.612171 23393 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:44480: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) I0627 19:31:54.612504 23402 master_main.cc:80] Initializing master server... I0627 19:31:54.616626 23402 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 226092us Jun 27 19:31:54 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153114, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:54.625305 23209 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } attempt: 2 I0627 19:31:54.626354 23402 init.cc:339] Logged in from keytab as kudu/127.12.131.60@KRBTEST.COM (short username kudu) I0627 19:31:54.627035 23429 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.626021 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:54.626469 (+ 448us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.626616 (+ 147us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":231} W0627 19:31:54.627547 23209 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:44480: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) I0627 19:31:54.642958 23402 fs_manager.cc:261] Metadata directory not provided I0627 19:31:54.643389 23402 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:54.643841 23402 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:54.644060 23402 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:54.660567 23402 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/data/instance: uuid: "6e88e61dad8a4be78044c758ab7bc93c" format_stamp: "Formatted at 2018-06-28 02:31:54 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:54.661947 23402 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/wal/instance: uuid: "6e88e61dad8a4be78044c758ab7bc93c" format_stamp: "Formatted at 2018-06-28 02:31:54 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:54.688819 23402 fs_manager.cc:501] Time spent creating directory manager: real 0.026s user 0.013s sys 0.008s I0627 19:31:54.691085 23402 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:54.692639 23402 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:54.703914 23209 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } attempt: 3 I0627 19:31:54.705760 23429 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.704732 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:54.705160 (+ 428us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.705307 (+ 147us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":233} W0627 19:31:54.706286 23209 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:44480: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) I0627 19:31:54.715205 23402 fs_manager.cc:417] Time spent opening block manager: real 0.009s user 0.001s sys 0.006s I0627 19:31:54.715416 23402 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/wal uuid: "6e88e61dad8a4be78044c758ab7bc93c" format_stamp: "Formatted at 2018-06-28 02:31:54 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:54.715842 23402 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:31:54.788489 23393 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } attempt: 4 I0627 19:31:54.790355 23398 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.789293 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:54.789775 (+ 482us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.789934 (+ 159us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":279} W0627 19:31:54.790879 23393 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:44480: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) I0627 19:31:54.844491 23209 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } attempt: 4 I0627 19:31:54.846344 23429 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:54.845265 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:54.845748 (+ 483us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:54.845906 (+ 158us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":272} W0627 19:31:54.846899 23209 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:44480: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) I0627 19:31:55.086515 23402 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:55.094272 23393 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } attempt: 5 I0627 19:31:55.096215 23398 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:55.095151 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:55.095627 (+ 476us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:55.095780 (+ 153us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":274} W0627 19:31:55.096743 23393 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:44480: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) I0627 19:31:55.097105 23402 master_main.cc:83] Starting Master server... I0627 19:31:55.130220 23209 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } attempt: 5 I0627 19:31:55.132107 23429 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:55.131054 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:55.131521 (+ 467us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:55.131671 (+ 150us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":241} W0627 19:31:55.134866 23209 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:44480: Network error: Client connection negotiation failed: client connection to 127.12.131.60:44480: connect: Connection refused (error 111) I0627 19:31:55.331635 23402 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.60:44480 I0627 19:31:55.332706 23402 webserver.cc:175] Starting webserver on 127.12.131.60:0 I0627 19:31:55.332815 23402 webserver.cc:186] Document root disabled I0627 19:31:55.334465 23402 webserver.cc:313] Webserver started. Bound to: http://127.12.131.60:36271/ I0627 19:31:55.335327 23402 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/master-2/data/info.pb I0627 19:31:55.338533 23558 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:31:55.341284 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 23402 I0627 19:31:55.351871 23558 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } has no permanent_uuid. Determining permanent_uuid... WARNING: no policy specified for kudu/127.12.131.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.1@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. I0627 19:31:55.381170 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.12.131.1:0 --local_ip_for_outbound_sockets=127.12.131.1 --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:51131,127.12.131.61:46323,127.12.131.60:44480 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_authentication=required --rpc_encryption=optional with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} Jun 27 19:31:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153114, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.60@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:31:55.403115 23560 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:55.355388 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:47962 0627 19:31:55.362095 (+ 6707us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:55.362121 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:55.375799 (+ 13678us) server_negotiation.cc:373] Connection header received 0627 19:31:55.380044 (+ 4245us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:55.380071 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:55.380194 (+ 123us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:55.380355 (+ 161us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:55.382781 (+ 2426us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.383144 (+ 363us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.384616 (+ 1472us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.385542 (+ 926us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.385704 (+ 162us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:55.389296 (+ 3592us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:55.389319 (+ 23us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:55.389342 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:55.389396 (+ 54us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:55.393450 (+ 4054us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:55.395192 (+ 1742us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:55.395215 (+ 23us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:55.395231 (+ 16us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:55.395792 (+ 561us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:55.396476 (+ 684us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:55.396498 (+ 22us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:55.396514 (+ 16us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:55.396933 (+ 419us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:55.397103 (+ 170us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:55.400698 (+ 3595us) server_negotiation.cc:264] Negotiation successful 0627 19:31:55.401227 (+ 529us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":6467,"thread_start_us":6236,"threads_started":1} I0627 19:31:55.404662 23561 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:55.355584 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:51131 0627 19:31:55.374992 (+ 19408us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:55.375662 (+ 670us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:55.379630 (+ 3968us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:55.380952 (+ 1322us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:55.380982 (+ 30us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:55.381832 (+ 850us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:55.382141 (+ 309us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.382525 (+ 384us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:55.382576 (+ 51us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.383450 (+ 874us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.383473 (+ 23us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.384367 (+ 894us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:55.384432 (+ 65us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.386266 (+ 1834us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.386288 (+ 22us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.386534 (+ 246us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:55.386704 (+ 170us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:55.386718 (+ 14us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:55.389018 (+ 2300us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:55.393669 (+ 4651us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:55.393700 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:55.393725 (+ 25us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:55.395005 (+ 1280us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:55.396017 (+ 1012us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:55.396040 (+ 23us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:55.396057 (+ 17us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:55.396282 (+ 225us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:55.397272 (+ 990us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:55.397301 (+ 29us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:55.397810 (+ 509us) client_negotiation.cc:695] Sending connection context 0627 19:31:55.398151 (+ 341us) client_negotiation.cc:225] Negotiation successful 0627 19:31:55.398774 (+ 623us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":19034,"thread_start_us":14432,"threads_started":1} I0627 19:31:55.406858 23558 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } has no permanent_uuid. Determining permanent_uuid... Jun 27 19:31:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153114, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.60@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM I0627 19:31:55.457455 23561 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:55.407737 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:46323 0627 19:31:55.415307 (+ 7570us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:55.415344 (+ 37us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:55.415516 (+ 172us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:55.424694 (+ 9178us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:55.424722 (+ 28us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:55.425225 (+ 503us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:55.425335 (+ 110us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.425482 (+ 147us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:55.425510 (+ 28us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.426247 (+ 737us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.426273 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.427083 (+ 810us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:55.427124 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.428710 (+ 1586us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.428735 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.428997 (+ 262us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:55.429159 (+ 162us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:55.429174 (+ 15us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:55.436718 (+ 7544us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:55.447605 (+ 10887us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:55.447632 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:55.447650 (+ 18us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:55.448775 (+ 1125us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:55.453478 (+ 4703us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:55.453504 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:55.453522 (+ 18us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:55.453682 (+ 160us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:55.454566 (+ 884us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:55.454588 (+ 22us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:55.454862 (+ 274us) client_negotiation.cc:695] Sending connection context 0627 19:31:55.455056 (+ 194us) client_negotiation.cc:225] Negotiation successful 0627 19:31:55.455566 (+ 510us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":7327} I0627 19:31:55.465873 23564 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:55.408500 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60568 0627 19:31:55.424097 (+ 15597us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:55.424121 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:55.424159 (+ 38us) server_negotiation.cc:373] Connection header received 0627 19:31:55.424266 (+ 107us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:55.424287 (+ 21us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:55.424430 (+ 143us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:55.424590 (+ 160us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:55.425654 (+ 1064us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.426056 (+ 402us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.427421 (+ 1365us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.428495 (+ 1074us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.437132 (+ 8637us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:55.437501 (+ 369us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:55.437526 (+ 25us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:55.437549 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:55.437618 (+ 69us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:55.447376 (+ 9758us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:55.452579 (+ 5203us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:55.452607 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:55.452623 (+ 16us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:55.453248 (+ 625us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:55.453902 (+ 654us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:55.453925 (+ 23us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:55.453941 (+ 16us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:55.454384 (+ 443us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:55.463403 (+ 9019us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:55.463663 (+ 260us) server_negotiation.cc:264] Negotiation successful 0627 19:31:55.464124 (+ 461us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":15381,"thread_start_us":15131,"threads_started":1} I0627 19:31:55.467658 23558 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } has no permanent_uuid. Determining permanent_uuid... Jun 27 19:31:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153114, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.60@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM I0627 19:31:55.525275 23561 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:55.468536 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:55.468809 (+ 273us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:55.468845 (+ 36us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:55.469001 (+ 156us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:55.492480 (+ 23479us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:55.492506 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:55.493236 (+ 730us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:55.493403 (+ 167us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.493576 (+ 173us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:55.493615 (+ 39us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.494655 (+ 1040us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.494681 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.495526 (+ 845us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:55.495569 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.504259 (+ 8690us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.504293 (+ 34us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.504578 (+ 285us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:55.504695 (+ 117us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:55.504706 (+ 11us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:55.506704 (+ 1998us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:55.516020 (+ 9316us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:55.516040 (+ 20us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:55.516052 (+ 12us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:55.517153 (+ 1101us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:55.518768 (+ 1615us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:55.518796 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:55.518816 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:55.518979 (+ 163us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:55.522210 (+ 3231us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:55.522237 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:55.522557 (+ 320us) client_negotiation.cc:695] Sending connection context 0627 19:31:55.522769 (+ 212us) client_negotiation.cc:225] Negotiation successful 0627 19:31:55.523269 (+ 500us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":75} I0627 19:31:55.532143 23567 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:55.469264 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:37920 0627 19:31:55.485783 (+ 16519us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:55.485815 (+ 32us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:55.485866 (+ 51us) server_negotiation.cc:373] Connection header received 0627 19:31:55.485998 (+ 132us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:55.486027 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:55.486229 (+ 202us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:55.492413 (+ 6184us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:55.493850 (+ 1437us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.494472 (+ 622us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.495839 (+ 1367us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.502162 (+ 6323us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.505753 (+ 3591us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:55.506991 (+ 1238us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:55.507029 (+ 38us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:55.507057 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:55.507130 (+ 73us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:55.515826 (+ 8696us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:55.517480 (+ 1654us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:55.517514 (+ 34us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:55.517534 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:55.518585 (+ 1051us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:55.519467 (+ 882us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:55.521364 (+ 1897us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:55.521407 (+ 43us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:55.522012 (+ 605us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:55.527280 (+ 5268us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:55.527577 (+ 297us) server_negotiation.cc:264] Negotiation successful 0627 19:31:55.530197 (+ 2620us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":16247,"thread_start_us":15866,"threads_started":1} I0627 19:31:55.540333 23558 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: Bootstrap starting. I0627 19:31:55.547935 23558 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: No blocks or log segments found. Creating new log. I0627 19:31:55.550411 23558 log.cc:525] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: Log is configured to *not* fsync() on all Append() calls I0627 19:31:55.572474 23558 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: No bootstrap required, opened a new log I0627 19:31:55.593585 23558 raft_consensus.cc:325] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } I0627 19:31:55.594555 23558 raft_consensus.cc:628] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e88e61dad8a4be78044c758ab7bc93c, State: Initialized, Role: FOLLOWER I0627 19:31:55.595078 23558 consensus_queue.cc:227] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } I0627 19:31:55.599711 23558 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:31:55.600865 23558 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [sys.catalog]: This master's current role is: FOLLOWER I0627 19:31:55.602236 23558 sys_catalog.cc:424] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [sys.catalog]: configured and running, proceeding with master startup. WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:55.626266 23563 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:55.627233 23563 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.1 W0627 19:31:55.627724 23563 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0627 19:31:55.631350 23393 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } attempt: 6 I0627 19:31:55.647224 23209 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } attempt: 6 W0627 19:31:55.664060 23563 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:55.664297 23563 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:55.664611 23563 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.1 W0627 19:31:55.665143 23563 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.1 Jun 27 19:31:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153113, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.61@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM I0627 19:31:55.698144 23575 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:55.632247 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:55.644111 (+ 11864us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:55.644155 (+ 44us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:55.647495 (+ 3340us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:55.648273 (+ 778us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:55.648297 (+ 24us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:55.649057 (+ 760us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:55.649224 (+ 167us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.649401 (+ 177us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:55.649437 (+ 36us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.650854 (+ 1417us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.650878 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.651669 (+ 791us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:55.651706 (+ 37us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.658835 (+ 7129us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.658866 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.659136 (+ 270us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:55.659294 (+ 158us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:55.659311 (+ 17us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:55.670002 (+ 10691us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:55.679889 (+ 9887us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:55.679918 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:55.679936 (+ 18us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:55.681172 (+ 1236us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:55.687181 (+ 6009us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:55.687207 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:55.687225 (+ 18us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:55.687410 (+ 185us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:55.692713 (+ 5303us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:55.692736 (+ 23us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:55.693024 (+ 288us) client_negotiation.cc:695] Sending connection context 0627 19:31:55.693231 (+ 207us) client_negotiation.cc:225] Negotiation successful 0627 19:31:55.696342 (+ 3111us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":11644,"thread_start_us":11870,"threads_started":1} I0627 19:31:55.695772 23567 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:55.633430 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:37922 0627 19:31:55.633727 (+ 297us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:55.633751 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:55.644236 (+ 10485us) server_negotiation.cc:373] Connection header received 0627 19:31:55.647709 (+ 3473us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:55.647735 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:55.647861 (+ 126us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:55.647986 (+ 125us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:55.650237 (+ 2251us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.650624 (+ 387us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.657093 (+ 6469us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.658180 (+ 1087us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.658358 (+ 178us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:55.670321 (+ 11963us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:55.670350 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:55.670372 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:55.670479 (+ 107us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:55.679610 (+ 9131us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:55.681358 (+ 1748us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:55.681408 (+ 50us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:55.681430 (+ 22us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:55.681996 (+ 566us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:55.687605 (+ 5609us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:55.687632 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:55.687649 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:55.688110 (+ 461us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:55.692587 (+ 4477us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:55.693630 (+ 1043us) server_negotiation.cc:264] Negotiation successful 0627 19:31:55.694099 (+ 469us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":87} I0627 19:31:55.706049 23393 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: Bootstrap starting. Jun 27 19:31:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153112, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM I0627 19:31:55.710115 23393 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: No blocks or log segments found. Creating new log. I0627 19:31:55.713742 23393 log.cc:525] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: Log is configured to *not* fsync() on all Append() calls I0627 19:31:55.717056 23563 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.1:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:51131,127.12.131.61:46323,127.12.131.60:44480 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:55.720729 23393 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: No bootstrap required, opened a new log I0627 19:31:55.738148 23576 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:55.648047 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:55.660704 (+ 12657us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:55.660744 (+ 40us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:55.660873 (+ 129us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:55.699634 (+ 38761us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:55.699662 (+ 28us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:55.700333 (+ 671us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:55.700525 (+ 192us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.700712 (+ 187us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:55.700750 (+ 38us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.701748 (+ 998us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.701772 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.702551 (+ 779us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:55.702589 (+ 38us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.707038 (+ 4449us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.707069 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:55.707335 (+ 266us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:55.707527 (+ 192us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:55.707543 (+ 16us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:55.710336 (+ 2793us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:55.725556 (+ 15220us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:55.725585 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:55.725605 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:55.726854 (+ 1249us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:55.729942 (+ 3088us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:55.729968 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:55.729988 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:55.730139 (+ 151us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:55.735105 (+ 4966us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:55.735134 (+ 29us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:55.735460 (+ 326us) client_negotiation.cc:695] Sending connection context 0627 19:31:55.735661 (+ 201us) client_negotiation.cc:225] Negotiation successful 0627 19:31:55.736180 (+ 519us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":12419,"thread_start_us":12127,"threads_started":1} I0627 19:31:55.741291 23567 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:55.649049 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:37923 0627 19:31:55.699030 (+ 49981us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:55.699054 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:55.699091 (+ 37us) server_negotiation.cc:373] Connection header received 0627 19:31:55.699188 (+ 97us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:55.699211 (+ 23us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:55.699340 (+ 129us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:55.700947 (+ 1607us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:55.701185 (+ 238us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.701563 (+ 378us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.702896 (+ 1333us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:55.705333 (+ 2437us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:55.710608 (+ 5275us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:55.710912 (+ 304us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:55.710937 (+ 25us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:55.710959 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:55.711029 (+ 70us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:55.724786 (+ 13757us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:55.727207 (+ 2421us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:55.727233 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:55.727252 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:55.727889 (+ 637us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:55.730356 (+ 2467us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:55.730406 (+ 50us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:55.730425 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:55.730873 (+ 448us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:55.738890 (+ 8017us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:55.739109 (+ 219us) server_negotiation.cc:264] Negotiation successful 0627 19:31:55.739569 (+ 460us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":49780,"threads_started":1} I0627 19:31:55.765179 23563 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:55.776609 23563 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 226592us Jun 27 19:31:55 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153115, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:55.781756 23563 init.cc:339] Logged in from keytab as kudu/127.12.131.1@KRBTEST.COM (short username kudu) I0627 19:31:55.785241 23393 raft_consensus.cc:325] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } I0627 19:31:55.785979 23393 raft_consensus.cc:628] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c68ec568384c83885614bc22ec1a50, State: Initialized, Role: FOLLOWER I0627 19:31:55.786485 23393 consensus_queue.cc:227] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } I0627 19:31:55.796229 23393 sys_catalog.cc:337] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:31:55.796967 23393 sys_catalog.cc:340] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [sys.catalog]: This master's current role is: FOLLOWER I0627 19:31:55.795843 23584 sys_catalog.cc:337] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:31:55.798365 23393 sys_catalog.cc:424] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:55.797297 23584 sys_catalog.cc:340] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [sys.catalog]: This master's current role is: FOLLOWER I0627 19:31:55.804479 23223 master_main.cc:86] Master server successfully started. I0627 19:31:55.807708 23563 fs_manager.cc:261] Metadata directory not provided I0627 19:31:55.807940 23563 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:55.808341 23563 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:55.808600 23563 server_base.cc:433] Attempting to create new FS layout instead W0627 19:31:55.825676 23586 catalog_manager.cc:1064] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:55.827147 23586 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:31:55.837297 23563 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/data/instance: uuid: "0d131be18f524b08b1559b91f6539f68" format_stamp: "Formatted at 2018-06-28 02:31:55 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:55.838495 23563 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/wal/instance: uuid: "0d131be18f524b08b1559b91f6539f68" format_stamp: "Formatted at 2018-06-28 02:31:55 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:55.600029 23573 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:31:55.865064 23573 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [sys.catalog]: This master's current role is: FOLLOWER I0627 19:31:55.879662 23563 fs_manager.cc:501] Time spent creating directory manager: real 0.040s user 0.014s sys 0.002s I0627 19:31:55.881592 23563 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:55.882205 23563 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:55.905884 23563 fs_manager.cc:417] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s I0627 19:31:55.906107 23563 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/wal uuid: "0d131be18f524b08b1559b91f6539f68" format_stamp: "Formatted at 2018-06-28 02:31:55 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:55.906499 23563 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0627 19:31:55.939393 23324 tablet.cc:1678] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: Can't schedule compaction. Clean time has not been advanced past its initial value. W0627 19:31:55.612293 23496 tablet.cc:1678] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:55.973954 23574 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0627 19:31:55.977027 23402 master_main.cc:86] Master server successfully started. I0627 19:31:55.982216 23209 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd: Bootstrap starting. I0627 19:31:55.985352 23209 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd: No blocks or log segments found. Creating new log. I0627 19:31:55.987538 23209 log.cc:525] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd: Log is configured to *not* fsync() on all Append() calls I0627 19:31:56.009030 23209 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd: No bootstrap required, opened a new log I0627 19:31:56.048171 23209 raft_consensus.cc:325] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } I0627 19:31:56.057606 23209 raft_consensus.cc:628] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d10270bf864c51bdcf2f696e252fdd, State: Initialized, Role: FOLLOWER I0627 19:31:56.059653 23209 consensus_queue.cc:227] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } I0627 19:31:56.069622 23598 sys_catalog.cc:337] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:31:56.070348 23598 sys_catalog.cc:340] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [sys.catalog]: This master's current role is: FOLLOWER I0627 19:31:56.070511 23209 sys_catalog.cc:337] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:31:56.080422 23209 sys_catalog.cc:340] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [sys.catalog]: This master's current role is: FOLLOWER I0627 19:31:56.083577 23209 sys_catalog.cc:424] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [sys.catalog]: configured and running, proceeding with master startup. I0627 19:31:56.088410 23599 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0627 19:31:56.093974 23061 master_main.cc:86] Master server successfully started. W0627 19:31:56.125490 23141 tablet.cc:1678] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:31:56.279213 23563 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:56.288022 23563 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:56.293838 23563 tablet_server_main.cc:90] Starting tablet server... W0627 19:31:56.829790 23586 catalog_manager.cc:1064] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:56.829999 23586 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:31:56.960188 23563 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.1:59030 I0627 19:31:56.961913 23563 webserver.cc:175] Starting webserver on 127.12.131.1:0 I0627 19:31:56.962110 23563 webserver.cc:186] Document root disabled I0627 19:31:56.968919 23563 webserver.cc:313] Webserver started. Bound to: http://127.12.131.1:38579/ I0627 19:31:56.969985 23563 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-0/data/info.pb I0627 19:31:56.976307 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 23563 W0627 19:31:56.983299 23574 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:56.983547 23574 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry WARNING: no policy specified for kudu/127.12.131.2@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.2@KRBTEST.COM" created. I0627 19:31:57.059972 23563 tablet_server_main.cc:93] Tablet server successfully started. Jun 27 19:31:57 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153115, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM Jun 27 19:31:57 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153115, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM Jun 27 19:31:57 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153115, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM W0627 19:31:57.098083 23599 catalog_manager.cc:1064] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:57.098306 23599 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. W0627 19:31:57.104012 12812 mini_kdc.cc:260] Time spent creating service keytab for kudu/127.12.131.2: real 0.127s user 0.002s sys 0.009s I0627 19:31:57.105607 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.12.131.2:0 --local_ip_for_outbound_sockets=127.12.131.2 --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:51131,127.12.131.61:46323,127.12.131.60:44480 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_authentication=required --rpc_encryption=optional with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:31:57.106580 23785 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:57.003892 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:57252 0627 19:31:57.014539 (+ 10647us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:57.014562 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:57.023252 (+ 8690us) server_negotiation.cc:373] Connection header received 0627 19:31:57.053565 (+ 30313us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:57.053593 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:57.053709 (+ 116us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:57.053848 (+ 139us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:57.059541 (+ 5693us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:57.060005 (+ 464us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:57.061777 (+ 1772us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:57.062841 (+ 1064us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:57.063015 (+ 174us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:57.074282 (+ 11267us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:57.074371 (+ 89us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:57.074424 (+ 53us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:57.074506 (+ 82us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:57.080815 (+ 6309us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:57.095603 (+ 14788us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:57.095632 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:57.095651 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:57.096278 (+ 627us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:57.100211 (+ 3933us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:57.100237 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:57.100254 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:57.100715 (+ 461us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:57.101100 (+ 385us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:57.104564 (+ 3464us) server_negotiation.cc:264] Negotiation successful 0627 19:31:57.104903 (+ 339us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":10390,"thread_start_us":10477,"threads_started":1} I0627 19:31:57.106997 23784 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:57.003544 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:46323 0627 19:31:57.022831 (+ 19287us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:57.023015 (+ 184us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:57.053331 (+ 30316us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:57.054057 (+ 726us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:57.054089 (+ 32us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:57.057784 (+ 3695us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:57.058591 (+ 807us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:57.059225 (+ 634us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:57.059295 (+ 70us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:57.060401 (+ 1106us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:57.060427 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:57.061526 (+ 1099us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:57.061580 (+ 54us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:57.064253 (+ 2673us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:57.064282 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:57.064641 (+ 359us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:57.064920 (+ 279us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:57.064938 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:57.073719 (+ 8781us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:57.084411 (+ 10692us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:57.084446 (+ 35us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:57.084474 (+ 28us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:57.095063 (+ 10589us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:57.099536 (+ 4473us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:57.099562 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:57.099580 (+ 18us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:57.099826 (+ 246us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:57.103334 (+ 3508us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:57.103389 (+ 55us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:57.103802 (+ 413us) client_negotiation.cc:695] Sending connection context 0627 19:31:57.104302 (+ 500us) client_negotiation.cc:225] Negotiation successful 0627 19:31:57.104715 (+ 413us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":15052,"mutex_wait_us":8536,"thread_start_us":5443,"threads_started":1} I0627 19:31:57.118050 23788 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:57.026106 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:59010 0627 19:31:57.047373 (+ 21267us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:57.047401 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:57.056622 (+ 9221us) server_negotiation.cc:373] Connection header received 0627 19:31:57.056861 (+ 239us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:57.056888 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:57.057004 (+ 116us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:57.057144 (+ 140us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:57.064620 (+ 7476us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:57.065065 (+ 445us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:57.068403 (+ 3338us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:57.069660 (+ 1257us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:57.069845 (+ 185us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:57.077391 (+ 7546us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:57.077531 (+ 140us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:57.077559 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:57.077646 (+ 87us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:57.084192 (+ 6546us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:57.097499 (+ 13307us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:57.097529 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:57.097547 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:57.098205 (+ 658us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:57.108240 (+ 10035us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:57.108265 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:57.108282 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:57.108745 (+ 463us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:57.109075 (+ 330us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:57.115782 (+ 6707us) server_negotiation.cc:264] Negotiation successful 0627 19:31:57.116150 (+ 368us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":21043,"thread_start_us":21284,"threads_started":1} I0627 19:31:57.121958 23782 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:56.985981 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:51131 0627 19:31:56.995403 (+ 9422us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:56.995507 (+ 104us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:57.052971 (+ 57464us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:57.055878 (+ 2907us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:57.057443 (+ 1565us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:57.060140 (+ 2697us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:57.062956 (+ 2816us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:57.063691 (+ 735us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:57.063729 (+ 38us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:57.068485 (+ 4756us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:57.068512 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:57.069290 (+ 778us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:57.069357 (+ 67us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:57.072027 (+ 2670us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:57.072052 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:57.072315 (+ 263us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:57.072581 (+ 266us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:57.072599 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:57.085436 (+ 12837us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:57.097645 (+ 12209us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:57.097673 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:57.097693 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:57.098981 (+ 1288us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:57.102440 (+ 3459us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:57.102466 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:57.102485 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:57.102705 (+ 220us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:57.114962 (+ 12257us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:57.114990 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:57.115282 (+ 292us) client_negotiation.cc:695] Sending connection context 0627 19:31:57.115890 (+ 608us) client_negotiation.cc:225] Negotiation successful 0627 19:31:57.116213 (+ 323us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":9067,"mutex_wait_us":4588,"thread_start_us":8045,"threads_started":1} I0627 19:31:57.123797 23787 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:57.025402 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:57.056452 (+ 31050us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:57.056525 (+ 73us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:57.056671 (+ 146us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:57.058119 (+ 1448us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:57.058752 (+ 633us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:57.062284 (+ 3532us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:57.062601 (+ 317us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:57.063945 (+ 1344us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:57.063984 (+ 39us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:57.066800 (+ 2816us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:57.066827 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:57.068184 (+ 1357us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:57.068225 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:57.071226 (+ 3001us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:57.071255 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:57.071553 (+ 298us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:57.071795 (+ 242us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:57.071811 (+ 16us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:57.076875 (+ 5064us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:57.095749 (+ 18874us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:57.095776 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:57.095797 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:57.096995 (+ 1198us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:57.107486 (+ 10491us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:57.107514 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:57.107536 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:57.107706 (+ 170us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:57.109455 (+ 1749us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:57.109478 (+ 23us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:57.109745 (+ 267us) client_negotiation.cc:695] Sending connection context 0627 19:31:57.119808 (+ 10063us) client_negotiation.cc:225] Negotiation successful 0627 19:31:57.121554 (+ 1746us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":30724,"mutex_wait_us":3155,"thread_start_us":32936,"threads_started":1} I0627 19:31:57.126358 23781 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:56.985776 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:35967 0627 19:31:56.997469 (+ 11693us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:56.997493 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:56.997531 (+ 38us) server_negotiation.cc:373] Connection header received 0627 19:31:57.053345 (+ 55814us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:57.053377 (+ 32us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:57.053529 (+ 152us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:57.053669 (+ 140us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:57.063939 (+ 10270us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:57.064345 (+ 406us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:57.069564 (+ 5219us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:57.070622 (+ 1058us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:57.070789 (+ 167us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:57.086095 (+ 15306us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:57.086176 (+ 81us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:57.086202 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:57.086280 (+ 78us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:57.092667 (+ 6387us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:57.100930 (+ 8263us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:57.100958 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:57.100976 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:57.101653 (+ 677us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:57.112028 (+ 10375us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:57.112057 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:57.112078 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:57.114328 (+ 2250us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:57.118884 (+ 4556us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:57.119289 (+ 405us) server_negotiation.cc:264] Negotiation successful 0627 19:31:57.119952 (+ 663us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":11476,"thread_start_us":7636,"threads_started":1} I0627 19:31:57.127215 23783 heartbeater.cc:346] Connected to a master server at 127.12.131.60:44480 I0627 19:31:57.127557 23783 heartbeater.cc:426] Registering TS with master... I0627 19:31:57.128216 23783 heartbeater.cc:475] Master 127.12.131.60:44480 requested a full tablet report, sending... I0627 19:31:57.130133 23777 heartbeater.cc:346] Connected to a master server at 127.12.131.62:51131 I0627 19:31:57.130386 23777 heartbeater.cc:426] Registering TS with master... I0627 19:31:57.130831 23777 heartbeater.cc:475] Master 127.12.131.62:51131 requested a full tablet report, sending... I0627 19:31:57.131870 23516 ts_manager.cc:98] Registered new tserver with Master: 0d131be18f524b08b1559b91f6539f68 (127.12.131.1:59030) I0627 19:31:57.134748 23158 ts_manager.cc:98] Registered new tserver with Master: 0d131be18f524b08b1559b91f6539f68 (127.12.131.1:59030) I0627 19:31:57.125030 23780 heartbeater.cc:346] Connected to a master server at 127.12.131.61:46323 I0627 19:31:57.191088 23780 heartbeater.cc:426] Registering TS with master... I0627 19:31:57.191624 23780 heartbeater.cc:475] Master 127.12.131.61:46323 requested a full tablet report, sending... I0627 19:31:57.194936 23340 ts_manager.cc:98] Registered new tserver with Master: 0d131be18f524b08b1559b91f6539f68 (127.12.131.1:59030) WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:31:57.527993 23797 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:31:57.528467 23797 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.2 W0627 19:31:57.528923 23797 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:31:57.561480 23797 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:31:57.570488 23797 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:31:57.570848 23797 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.2 W0627 19:31:57.571466 23797 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.2 I0627 19:31:57.638662 23797 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.2:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:51131,127.12.131.61:46323,127.12.131.60:44480 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.2 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:31:57.662684 23797 tablet_server_main.cc:85] Initializing tablet server... I0627 19:31:57.673812 23797 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 227592us Jun 27 19:31:57 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153117, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:31:57.677248 23797 init.cc:339] Logged in from keytab as kudu/127.12.131.2@KRBTEST.COM (short username kudu) I0627 19:31:57.689844 23797 fs_manager.cc:261] Metadata directory not provided I0627 19:31:57.690095 23797 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:31:57.690556 23797 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:31:57.690788 23797 server_base.cc:433] Attempting to create new FS layout instead I0627 19:31:57.718369 23797 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/data/instance: uuid: "492e246a82c04f9b97b86bd9893d380e" format_stamp: "Formatted at 2018-06-28 02:31:57 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:57.728286 23797 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/wal/instance: uuid: "492e246a82c04f9b97b86bd9893d380e" format_stamp: "Formatted at 2018-06-28 02:31:57 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:57.759860 23797 fs_manager.cc:501] Time spent creating directory manager: real 0.031s user 0.013s sys 0.002s I0627 19:31:57.761864 23797 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:31:57.762617 23797 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:31:57.786152 23797 fs_manager.cc:417] Time spent opening block manager: real 0.008s user 0.000s sys 0.002s I0627 19:31:57.786430 23797 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/wal uuid: "492e246a82c04f9b97b86bd9893d380e" format_stamp: "Formatted at 2018-06-28 02:31:57 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:31:57.786834 23797 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0627 19:31:57.832465 23586 catalog_manager.cc:1064] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:57.832674 23586 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:31:57.985905 23574 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:57.986107 23574 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:31:58.100859 23599 catalog_manager.cc:1064] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:58.101080 23599 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:31:58.391100 23797 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:31:58.401640 23797 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:31:58.408888 23797 tablet_server_main.cc:90] Starting tablet server... W0627 19:31:58.835091 23586 catalog_manager.cc:1064] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:58.835297 23586 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:31:58.988585 23574 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:58.988803 23574 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:31:59.103437 23599 catalog_manager.cc:1064] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:59.103642 23599 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:31:59.610572 23797 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.2:34097 I0627 19:31:59.612509 23797 webserver.cc:175] Starting webserver on 127.12.131.2:0 I0627 19:31:59.612677 23797 webserver.cc:186] Document root disabled I0627 19:31:59.620551 23797 webserver.cc:313] Webserver started. Bound to: http://127.12.131.2:49083/ I0627 19:31:59.621599 23797 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-1/data/info.pb I0627 19:31:59.623709 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 23797 WARNING: no policy specified for kudu/127.12.131.3@KRBTEST.COM; defaulting to no policy I0627 19:31:59.679630 23797 tablet_server_main.cc:93] Tablet server successfully started. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.3@KRBTEST.COM" created. Jun 27 19:31:59 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153117, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Jun 27 19:31:59 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153117, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM W0627 19:31:59.741228 12812 mini_kdc.cc:260] Time spent creating service keytab for kudu/127.12.131.3: real 0.117s user 0.001s sys 0.011s I0627 19:31:59.742846 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.12.131.3:0 --local_ip_for_outbound_sockets=127.12.131.3 --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:51131,127.12.131.61:46323,127.12.131.60:44480 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_authentication=required --rpc_encryption=optional with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} Jun 27 19:31:59 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153117, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM I0627 19:31:59.771920 24041 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:59.648763 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:52669 0627 19:31:59.666843 (+ 18080us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:59.666865 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:59.677151 (+ 10286us) server_negotiation.cc:373] Connection header received 0627 19:31:59.712476 (+ 35325us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:59.712517 (+ 41us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:59.712662 (+ 145us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:59.712845 (+ 183us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:59.722137 (+ 9292us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:59.722581 (+ 444us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:59.726112 (+ 3531us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:59.727197 (+ 1085us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:59.727397 (+ 200us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:59.741262 (+ 13865us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:59.741347 (+ 85us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:59.741373 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:59.741484 (+ 111us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:59.748238 (+ 6754us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:59.761529 (+ 13291us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:59.761555 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:59.761570 (+ 15us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:59.762171 (+ 601us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:59.764291 (+ 2120us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:59.764317 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:59.764335 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:59.764830 (+ 495us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:59.765163 (+ 333us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:59.769835 (+ 4672us) server_negotiation.cc:264] Negotiation successful 0627 19:31:59.770200 (+ 365us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":17869,"thread_start_us":17600,"threads_started":1} I0627 19:31:59.772074 24044 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:59.653917 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:31:59.683685 (+ 29768us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:59.683839 (+ 154us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:59.712217 (+ 28378us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:59.713448 (+ 1231us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:59.713482 (+ 34us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:59.715028 (+ 1546us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:59.719368 (+ 4340us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:59.719962 (+ 594us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:59.720013 (+ 51us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:59.721259 (+ 1246us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:59.721291 (+ 32us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:59.725011 (+ 3720us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:59.725062 (+ 51us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:59.727811 (+ 2749us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:59.727839 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:59.728159 (+ 320us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:59.729197 (+ 1038us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:59.729597 (+ 400us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:59.744198 (+ 14601us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:59.752702 (+ 8504us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:59.752729 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:59.752752 (+ 23us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:59.760414 (+ 7662us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:59.762930 (+ 2516us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:59.762960 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:59.762980 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:59.763316 (+ 336us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:59.768691 (+ 5375us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:59.768718 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:59.769026 (+ 308us) client_negotiation.cc:695] Sending connection context 0627 19:31:59.769426 (+ 400us) client_negotiation.cc:225] Negotiation successful 0627 19:31:59.769754 (+ 328us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":29503,"mutex_wait_us":18866,"thread_start_us":8459,"threads_started":1} I0627 19:31:59.775060 24045 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:59.654294 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:33354 0627 19:31:59.670964 (+ 16670us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:59.670989 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:59.683963 (+ 12974us) server_negotiation.cc:373] Connection header received 0627 19:31:59.712513 (+ 28550us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:59.712545 (+ 32us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:59.712671 (+ 126us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:59.712817 (+ 146us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:59.720379 (+ 7562us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:59.720869 (+ 490us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:59.725981 (+ 5112us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:59.727113 (+ 1132us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:59.727291 (+ 178us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:59.744907 (+ 17616us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:59.744975 (+ 68us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:59.744997 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:59.745076 (+ 79us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:59.752065 (+ 6989us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:59.761461 (+ 9396us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:59.761492 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:59.761511 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:59.762174 (+ 663us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:59.764324 (+ 2150us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:59.764377 (+ 53us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:59.764396 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:59.764849 (+ 453us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:59.765179 (+ 330us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:59.772781 (+ 7602us) server_negotiation.cc:264] Negotiation successful 0627 19:31:59.773150 (+ 369us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":16452,"thread_start_us":7594,"threads_started":1} I0627 19:31:59.776474 24046 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:59.650471 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:46323 0627 19:31:59.672071 (+ 21600us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:59.676871 (+ 4800us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:59.711926 (+ 35055us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:59.713469 (+ 1543us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:59.713499 (+ 30us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:59.718108 (+ 4609us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:59.721506 (+ 3398us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:59.721875 (+ 369us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:59.721917 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:59.723035 (+ 1118us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:59.723061 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:59.725685 (+ 2624us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:59.725730 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:59.728174 (+ 2444us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:59.728203 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:59.728552 (+ 349us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:59.728890 (+ 338us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:59.728911 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:59.739150 (+ 10239us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:59.749071 (+ 9921us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:59.749105 (+ 34us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:59.749135 (+ 30us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:59.760922 (+ 11787us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:59.762868 (+ 1946us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:59.762895 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:59.762915 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:59.763552 (+ 637us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:59.765561 (+ 2009us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:59.765592 (+ 31us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:59.766099 (+ 507us) client_negotiation.cc:695] Sending connection context 0627 19:31:59.766644 (+ 545us) client_negotiation.cc:225] Negotiation successful 0627 19:31:59.767065 (+ 421us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":20814,"mutex_wait_us":3300,"thread_start_us":21390,"threads_started":1} I0627 19:31:59.778036 24040 heartbeater.cc:346] Connected to a master server at 127.12.131.60:44480 I0627 19:31:59.778358 24040 heartbeater.cc:426] Registering TS with master... I0627 19:31:59.779323 24040 heartbeater.cc:475] Master 127.12.131.60:44480 requested a full tablet report, sending... I0627 19:31:59.782519 24038 heartbeater.cc:346] Connected to a master server at 127.12.131.61:46323 I0627 19:31:59.782763 24038 heartbeater.cc:426] Registering TS with master... I0627 19:31:59.783318 24038 heartbeater.cc:475] Master 127.12.131.61:46323 requested a full tablet report, sending... I0627 19:31:59.783646 23516 ts_manager.cc:98] Registered new tserver with Master: 492e246a82c04f9b97b86bd9893d380e (127.12.131.2:34097) I0627 19:31:59.786310 24037 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:59.633135 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:35421 0627 19:31:59.637547 (+ 4412us) server_negotiation.cc:176] Beginning negotiation 0627 19:31:59.637572 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:31:59.648443 (+ 10871us) server_negotiation.cc:373] Connection header received 0627 19:31:59.715301 (+ 66858us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:31:59.715334 (+ 33us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:31:59.715489 (+ 155us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:31:59.715635 (+ 146us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:31:59.729659 (+ 14024us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:59.730110 (+ 451us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:59.734348 (+ 4238us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:31:59.735507 (+ 1159us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:31:59.735701 (+ 194us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:59.763580 (+ 27879us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:31:59.763665 (+ 85us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:31:59.763692 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:31:59.763779 (+ 87us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:31:59.770476 (+ 6697us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:59.775135 (+ 4659us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:59.775165 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:59.775357 (+ 192us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:59.776134 (+ 777us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:31:59.777328 (+ 1194us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:31:59.777354 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:31:59.777373 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:31:59.777843 (+ 470us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:31:59.782716 (+ 4873us) server_negotiation.cc:901] Waiting for connection context 0627 19:31:59.783168 (+ 452us) server_negotiation.cc:264] Negotiation successful 0627 19:31:59.784511 (+ 1343us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":4170,"thread_start_us":3940,"threads_started":1} I0627 19:31:59.786607 23340 ts_manager.cc:98] Registered new tserver with Master: 492e246a82c04f9b97b86bd9893d380e (127.12.131.2:34097) W0627 19:31:59.837736 23586 catalog_manager.cc:1064] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:59.837947 23586 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:31:59.783782 24036 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:31:59.631884 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:51131 0627 19:31:59.647804 (+ 15920us) negotiation.cc:98] Waiting for socket to connect 0627 19:31:59.647932 (+ 128us) client_negotiation.cc:160] Beginning negotiation 0627 19:31:59.711926 (+ 63994us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:31:59.719031 (+ 7105us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:31:59.719138 (+ 107us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:31:59.724248 (+ 5110us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:31:59.724481 (+ 233us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:59.729413 (+ 4932us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:59.729455 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:59.733042 (+ 3587us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:59.733070 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:59.734109 (+ 1039us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:31:59.734152 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:31:59.735732 (+ 1580us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:31:59.735773 (+ 41us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:31:59.736089 (+ 316us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:31:59.736453 (+ 364us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:31:59.736475 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:31:59.761023 (+ 24548us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:31:59.772621 (+ 11598us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:59.772652 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:59.772671 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:59.773911 (+ 1240us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:59.776704 (+ 2793us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:31:59.776731 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:31:59.776751 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:31:59.776935 (+ 184us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:31:59.778684 (+ 1749us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:31:59.778714 (+ 30us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:31:59.779120 (+ 406us) client_negotiation.cc:695] Sending connection context 0627 19:31:59.779613 (+ 493us) client_negotiation.cc:225] Negotiation successful 0627 19:31:59.780011 (+ 398us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":15341,"mutex_wait_us":16546,"spinlock_wait_cycles":95104,"thread_start_us":4794,"threads_started":1} I0627 19:31:59.842492 24033 heartbeater.cc:346] Connected to a master server at 127.12.131.62:51131 I0627 19:31:59.842743 24033 heartbeater.cc:426] Registering TS with master... I0627 19:31:59.843189 24033 heartbeater.cc:475] Master 127.12.131.62:51131 requested a full tablet report, sending... I0627 19:31:59.845613 23158 ts_manager.cc:98] Registered new tserver with Master: 492e246a82c04f9b97b86bd9893d380e (127.12.131.2:34097) W0627 19:31:59.991312 23574 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:31:59.991541 23574 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:00.106005 23599 catalog_manager.cc:1064] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:00.106220 23599 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:32:00.124665 24056 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:32:00.125149 24056 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.3 W0627 19:32:00.125586 24056 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:32:00.152495 24056 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:32:00.158089 24056 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:32:00.158462 24056 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.3 W0627 19:32:00.159061 24056 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.3 I0627 19:32:00.253058 24056 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.3:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:51131,127.12.131.61:46323,127.12.131.60:44480 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.3 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:32:00.276006 24056 tablet_server_main.cc:85] Initializing tablet server... I0627 19:32:00.280977 24056 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 229092us Jun 27 19:32:00 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153120, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:32:00.284487 24056 init.cc:339] Logged in from keytab as kudu/127.12.131.3@KRBTEST.COM (short username kudu) I0627 19:32:00.296550 24056 fs_manager.cc:261] Metadata directory not provided I0627 19:32:00.296802 24056 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:32:00.297276 24056 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:32:00.297677 24056 server_base.cc:433] Attempting to create new FS layout instead I0627 19:32:00.327555 24056 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/data/instance: uuid: "9b9b9080fb574dc3b65a74012c8fbd05" format_stamp: "Formatted at 2018-06-28 02:32:00 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:00.337302 24056 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/wal/instance: uuid: "9b9b9080fb574dc3b65a74012c8fbd05" format_stamp: "Formatted at 2018-06-28 02:32:00 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:00.377946 24056 fs_manager.cc:501] Time spent creating directory manager: real 0.040s user 0.014s sys 0.003s I0627 19:32:00.379936 24056 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:32:00.380642 24056 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:32:00.430599 24056 fs_manager.cc:417] Time spent opening block manager: real 0.032s user 0.004s sys 0.008s I0627 19:32:00.430788 24056 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/wal uuid: "9b9b9080fb574dc3b65a74012c8fbd05" format_stamp: "Formatted at 2018-06-28 02:32:00 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:00.431120 24056 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0627 19:32:00.840265 23586 catalog_manager.cc:1064] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:00.840492 23586 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:32:00.876420 24056 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:32:00.891078 24056 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:32:00.896730 24056 tablet_server_main.cc:90] Starting tablet server... W0627 19:32:00.993888 23574 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:00.994089 23574 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:01.108465 23599 catalog_manager.cc:1064] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:01.108681 23599 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:32:01.569723 24056 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.3:42814 I0627 19:32:01.571465 24056 webserver.cc:175] Starting webserver on 127.12.131.3:0 I0627 19:32:01.571656 24056 webserver.cc:186] Document root disabled I0627 19:32:01.578320 24056 webserver.cc:313] Webserver started. Bound to: http://127.12.131.3:60920/ I0627 19:32:01.579280 24056 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1530153042096250-12812/minicluster-data/ts-2/data/info.pb I0627 19:32:01.586079 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 24056 Jun 27 19:32:01 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM I0627 19:32:01.630276 24056 tablet_server_main.cc:93] Tablet server successfully started. I0627 19:32:01.635738 24306 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:01.588185 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:37936 0627 19:32:01.601706 (+ 13521us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:01.601738 (+ 32us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:01.601777 (+ 39us) server_negotiation.cc:373] Connection header received 0627 19:32:01.601889 (+ 112us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:01.601915 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:01.602040 (+ 125us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:01.602183 (+ 143us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:01.606526 (+ 4343us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.606961 (+ 435us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.611565 (+ 4604us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.612756 (+ 1191us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.613040 (+ 284us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:01.618587 (+ 5547us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:01.618615 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:01.618636 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:01.618718 (+ 82us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:01.624780 (+ 6062us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.628430 (+ 3650us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.628481 (+ 51us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.628502 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.629101 (+ 599us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.629721 (+ 620us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.629746 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.629763 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.630222 (+ 459us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:01.630621 (+ 399us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:01.632550 (+ 1929us) server_negotiation.cc:264] Negotiation successful 0627 19:32:01.633023 (+ 473us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":12821,"thread_start_us":21192,"threads_started":1} Jun 27 19:32:01 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM Jun 27 19:32:01 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153120, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM I0627 19:32:01.685642 24316 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:01.639504 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60588 0627 19:32:01.650252 (+ 10748us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:01.650276 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:01.650318 (+ 42us) server_negotiation.cc:373] Connection header received 0627 19:32:01.650430 (+ 112us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:01.650477 (+ 47us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:01.650600 (+ 123us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:01.653400 (+ 2800us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:01.654909 (+ 1509us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.655355 (+ 446us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.656809 (+ 1454us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.659251 (+ 2442us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.664953 (+ 5702us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:01.665312 (+ 359us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:01.665341 (+ 29us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:01.665364 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:01.665852 (+ 488us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:01.674411 (+ 8559us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.677197 (+ 2786us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.677226 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.677246 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.677938 (+ 692us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.678588 (+ 650us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.678618 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.678638 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.679122 (+ 484us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:01.682945 (+ 3823us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:01.683227 (+ 282us) server_negotiation.cc:264] Negotiation successful 0627 19:32:01.683781 (+ 554us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":10528,"thread_start_us":11592,"threads_started":1} Jun 27 19:32:01 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153120, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM Jun 27 19:32:01 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153120, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM I0627 19:32:01.725183 24304 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:01.589282 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:46323 0627 19:32:01.627603 (+ 38321us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:01.627795 (+ 192us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:01.652076 (+ 24281us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:01.654535 (+ 2459us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:01.654570 (+ 35us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:01.658363 (+ 3793us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:01.658716 (+ 353us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:01.661525 (+ 2809us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:01.661568 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.662501 (+ 933us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.662527 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:01.663670 (+ 1143us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:01.663719 (+ 49us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.668315 (+ 4596us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.668347 (+ 32us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:01.668720 (+ 373us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:01.669028 (+ 308us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:01.669052 (+ 24us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:01.686164 (+ 17112us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:01.705393 (+ 19229us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:01.705462 (+ 69us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:01.705491 (+ 29us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:01.708320 (+ 2829us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:01.714021 (+ 5701us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:01.714050 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:01.714072 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:01.714380 (+ 308us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:01.716173 (+ 1793us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:01.716202 (+ 29us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:01.716672 (+ 470us) client_negotiation.cc:695] Sending connection context 0627 19:32:01.717175 (+ 503us) client_negotiation.cc:225] Negotiation successful 0627 19:32:01.717606 (+ 431us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":37993,"mutex_wait_us":1206,"thread_start_us":9442,"threads_started":1} I0627 19:32:01.728324 24303 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:01.583286 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:33914 0627 19:32:01.596837 (+ 13551us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:01.596861 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:01.600497 (+ 3636us) server_negotiation.cc:373] Connection header received 0627 19:32:01.653615 (+ 53118us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:01.653649 (+ 34us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:01.653774 (+ 125us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:01.653915 (+ 141us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:01.668000 (+ 14085us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.668495 (+ 495us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.676952 (+ 8457us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.678036 (+ 1084us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.678203 (+ 167us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:01.697029 (+ 18826us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:01.697102 (+ 73us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:01.697131 (+ 29us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:01.697211 (+ 80us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:01.704238 (+ 7027us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.717777 (+ 13539us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.717806 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.717826 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.718531 (+ 705us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.721025 (+ 2494us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.721055 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.721076 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.721574 (+ 498us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:01.722042 (+ 468us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:01.726069 (+ 4027us) server_negotiation.cc:264] Negotiation successful 0627 19:32:01.726429 (+ 360us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":13334,"thread_start_us":16034,"threads_started":1} I0627 19:32:01.728909 24308 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:01.589635 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:60338 0627 19:32:01.600504 (+ 10869us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:01.600530 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:01.627934 (+ 27404us) server_negotiation.cc:373] Connection header received 0627 19:32:01.654050 (+ 26116us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:01.654077 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:01.654187 (+ 110us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:01.654317 (+ 130us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:01.661748 (+ 7431us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.662277 (+ 529us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.665654 (+ 3377us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.666828 (+ 1174us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.667017 (+ 189us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:01.697224 (+ 30207us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:01.697314 (+ 90us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:01.697337 (+ 23us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:01.697445 (+ 108us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:01.704531 (+ 7086us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.710511 (+ 5980us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.710538 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.710555 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.711197 (+ 642us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.714945 (+ 3748us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.714971 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.714990 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.715492 (+ 502us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:01.715832 (+ 340us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:01.718083 (+ 2251us) server_negotiation.cc:264] Negotiation successful 0627 19:32:01.718459 (+ 376us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":10606,"thread_start_us":10765,"threads_started":1} I0627 19:32:01.728256 24309 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:01.584084 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:51131 0627 19:32:01.600283 (+ 16199us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:01.600378 (+ 95us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:01.651761 (+ 51383us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:01.659957 (+ 8196us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:01.660070 (+ 113us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:01.665096 (+ 5026us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:01.666619 (+ 1523us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:01.666941 (+ 322us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:01.666983 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.669835 (+ 2852us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.669862 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:01.676658 (+ 6796us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:01.676709 (+ 51us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.679554 (+ 2845us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.679582 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:01.681591 (+ 2009us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:01.682503 (+ 912us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:01.682524 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:01.696470 (+ 13946us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:01.704912 (+ 8442us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:01.704946 (+ 34us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:01.704977 (+ 31us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:01.712676 (+ 7699us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:01.718948 (+ 6272us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:01.718972 (+ 24us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:01.718989 (+ 17us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:01.719142 (+ 153us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:01.722009 (+ 2867us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:01.722036 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:01.722341 (+ 305us) client_negotiation.cc:695] Sending connection context 0627 19:32:01.725789 (+ 3448us) client_negotiation.cc:225] Negotiation successful 0627 19:32:01.726117 (+ 328us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":15699,"mutex_wait_us":12559,"thread_start_us":46978,"threads_started":1} I0627 19:32:01.732095 24302 heartbeater.cc:346] Connected to a master server at 127.12.131.61:46323 I0627 19:32:01.732488 24302 heartbeater.cc:426] Registering TS with master... I0627 19:32:01.733242 24302 heartbeater.cc:475] Master 127.12.131.61:46323 requested a full tablet report, sending... I0627 19:32:01.734973 24301 heartbeater.cc:346] Connected to a master server at 127.12.131.62:51131 I0627 19:32:01.735213 24301 heartbeater.cc:426] Registering TS with master... I0627 19:32:01.735673 24301 heartbeater.cc:475] Master 127.12.131.62:51131 requested a full tablet report, sending... I0627 19:32:01.736038 23340 ts_manager.cc:98] Registered new tserver with Master: 9b9b9080fb574dc3b65a74012c8fbd05 (127.12.131.3:42814) Jun 27 19:32:01 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[23043](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:32:01.738919 23158 ts_manager.cc:98] Registered new tserver with Master: 9b9b9080fb574dc3b65a74012c8fbd05 (127.12.131.3:42814) I0627 19:32:01.743952 24315 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:01.635388 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:44480 0627 19:32:01.653059 (+ 17671us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:01.653211 (+ 152us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:01.653375 (+ 164us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:01.654319 (+ 944us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:01.654355 (+ 36us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:01.659697 (+ 5342us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:01.660312 (+ 615us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:01.660785 (+ 473us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:01.660831 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.663982 (+ 3151us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.664010 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:01.680115 (+ 16105us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:01.680161 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.682738 (+ 2577us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.682766 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:01.683863 (+ 1097us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:01.695693 (+ 11830us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:01.695715 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:01.712257 (+ 16542us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:01.722929 (+ 10672us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:01.722960 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:01.722982 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:01.724340 (+ 1358us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:01.732136 (+ 7796us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:01.732165 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:01.732189 (+ 24us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:01.732391 (+ 202us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:01.740660 (+ 8269us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:01.740686 (+ 26us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:01.741008 (+ 322us) client_negotiation.cc:695] Sending connection context 0627 19:32:01.741405 (+ 397us) client_negotiation.cc:225] Negotiation successful 0627 19:32:01.741735 (+ 330us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":17236,"mutex_wait_us":20657,"thread_start_us":17929,"threads_started":1} I0627 19:32:01.744308 24306 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:01.632786 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:42574 0627 19:32:01.636584 (+ 3798us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:01.636605 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:01.653315 (+ 16710us) server_negotiation.cc:373] Connection header received 0627 19:32:01.653575 (+ 260us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:01.653600 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:01.653722 (+ 122us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:01.653845 (+ 123us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:01.662409 (+ 8564us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.662822 (+ 413us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.680850 (+ 18028us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.681897 (+ 1047us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.682102 (+ 205us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:01.712807 (+ 30705us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:01.712883 (+ 76us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:01.712908 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:01.712991 (+ 83us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:01.722232 (+ 9241us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.724967 (+ 2735us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.724996 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.725015 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.731501 (+ 6486us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.736212 (+ 4711us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.736240 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.736259 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.739231 (+ 2972us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:01.739616 (+ 385us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:01.742033 (+ 2417us) server_negotiation.cc:264] Negotiation successful 0627 19:32:01.742522 (+ 489us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":3588,"thread_start_us":1517,"threads_started":1} I0627 19:32:01.751804 24307 heartbeater.cc:346] Connected to a master server at 127.12.131.60:44480 I0627 19:32:01.752066 24307 heartbeater.cc:426] Registering TS with master... I0627 19:32:01.752589 24307 heartbeater.cc:475] Master 127.12.131.60:44480 requested a full tablet report, sending... I0627 19:32:01.755043 23516 ts_manager.cc:98] Registered new tserver with Master: 9b9b9080fb574dc3b65a74012c8fbd05 (127.12.131.3:42814) I0627 19:32:01.760697 24319 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:01.689527 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:47985 0627 19:32:01.707306 (+ 17779us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:01.707330 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:01.707369 (+ 39us) server_negotiation.cc:373] Connection header received 0627 19:32:01.707508 (+ 139us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:01.707534 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:01.707657 (+ 123us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:01.709915 (+ 2258us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:01.710188 (+ 273us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.710792 (+ 604us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.712278 (+ 1486us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.730045 (+ 17767us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.735735 (+ 5690us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:01.737848 (+ 2113us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:01.737879 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:01.737912 (+ 33us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:01.737999 (+ 87us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:01.750686 (+ 12687us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.752806 (+ 2120us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.752836 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.752856 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.754136 (+ 1280us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.754799 (+ 663us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.754826 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.754846 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.755322 (+ 476us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:01.758104 (+ 2782us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:01.758321 (+ 217us) server_negotiation.cc:264] Negotiation successful 0627 19:32:01.758801 (+ 480us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":17554,"thread_start_us":17258,"threads_started":1} I0627 19:32:01.767931 12812 external_mini_cluster.cc:490] 3 TS(s) registered with all masters I0627 19:32:01.769520 23173 tablet_service.cc:1112] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000" dest_uuid: "b9d10270bf864c51bdcf2f696e252fdd" from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.12.131.62:47985 I0627 19:32:01.770015 23173 raft_consensus.cc:457] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0627 19:32:01.770293 23173 raft_consensus.cc:2782] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [term 0 FOLLOWER]: Advancing to term 1 I0627 19:32:01.775004 23173 raft_consensus.cc:479] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } I0627 19:32:01.776402 23173 leader_election.cc:243] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [CANDIDATE]: Term 1 election: Requesting vote from peer c7c68ec568384c83885614bc22ec1a50 (127.12.131.61:46323) I0627 19:32:01.778506 23173 leader_election.cc:243] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [CANDIDATE]: Term 1 election: Requesting vote from peer 6e88e61dad8a4be78044c758ab7bc93c (127.12.131.60:44480) I0627 19:32:01.778497 23359 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9d10270bf864c51bdcf2f696e252fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c7c68ec568384c83885614bc22ec1a50" I0627 19:32:01.779095 23359 raft_consensus.cc:2782] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:32:01.780167 23529 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9d10270bf864c51bdcf2f696e252fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6e88e61dad8a4be78044c758ab7bc93c" I0627 19:32:01.780755 23529 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [term 0 FOLLOWER]: Advancing to term 1 I0627 19:32:01.783993 23359 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d10270bf864c51bdcf2f696e252fdd in term 1. I0627 19:32:01.785156 23130 leader_election.cc:379] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [CANDIDATE]: Term 1 election: Vote granted by peer c7c68ec568384c83885614bc22ec1a50 (127.12.131.61:46323) I0627 19:32:01.785434 23130 leader_election.cc:270] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:32:01.785405 23529 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d10270bf864c51bdcf2f696e252fdd in term 1. I0627 19:32:01.786201 23136 leader_election.cc:379] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [CANDIDATE]: Term 1 election: Vote granted by peer 6e88e61dad8a4be78044c758ab7bc93c (127.12.131.60:44480) I0627 19:32:01.790623 24320 raft_consensus.cc:2537] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:32:01.805337 24320 raft_consensus.cc:593] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [term 1 LEADER]: Becoming Leader. State: Replica: b9d10270bf864c51bdcf2f696e252fdd, State: Running, Role: LEADER I0627 19:32:01.806056 24320 consensus_queue.cc:204] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } I0627 19:32:01.818554 24322 sys_catalog.cc:337] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9d10270bf864c51bdcf2f696e252fdd. Latest consensus state: current_term: 1 leader_uuid: "b9d10270bf864c51bdcf2f696e252fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:32:01.819229 24322 sys_catalog.cc:340] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [sys.catalog]: This master's current role is: LEADER I0627 19:32:01.824007 24329 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:32:01.831595 24329 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... W0627 19:32:01.842831 23586 catalog_manager.cc:1064] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:01.843041 23586 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:32:01.879329 24306 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:01.828953 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:37942 0627 19:32:01.829248 (+ 295us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:01.829272 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:01.849273 (+ 20001us) server_negotiation.cc:373] Connection header received 0627 19:32:01.849706 (+ 433us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:01.849731 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:01.849859 (+ 128us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:01.849982 (+ 123us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:01.853359 (+ 3377us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.853801 (+ 442us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.855298 (+ 1497us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.856352 (+ 1054us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.856548 (+ 196us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:01.865373 (+ 8825us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:01.865401 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:01.865423 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:01.865525 (+ 102us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:01.871724 (+ 6199us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.873550 (+ 1826us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.873575 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.873592 (+ 17us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.874173 (+ 581us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.874756 (+ 583us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.874781 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.874799 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.875249 (+ 450us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:01.875554 (+ 305us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:01.877080 (+ 1526us) server_negotiation.cc:264] Negotiation successful 0627 19:32:01.877543 (+ 463us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":88} I0627 19:32:01.886646 24308 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:01.816062 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60591 0627 19:32:01.816438 (+ 376us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:01.816463 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:01.825151 (+ 8688us) server_negotiation.cc:373] Connection header received 0627 19:32:01.825383 (+ 232us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:01.825434 (+ 51us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:01.825568 (+ 134us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:01.825699 (+ 131us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:01.836113 (+ 10414us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.836573 (+ 460us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.840409 (+ 3836us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.841484 (+ 1075us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.841651 (+ 167us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:01.851899 (+ 10248us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:01.851930 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:01.851952 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:01.852040 (+ 88us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:01.860674 (+ 8634us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.874529 (+ 13855us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.874561 (+ 32us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.874581 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.875248 (+ 667us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.881547 (+ 6299us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.881579 (+ 32us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.881599 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.882123 (+ 524us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:01.884043 (+ 1920us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:01.884280 (+ 237us) server_negotiation.cc:264] Negotiation successful 0627 19:32:01.884809 (+ 529us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":144} I0627 19:32:01.901980 24319 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:01.817508 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:47987 0627 19:32:01.817908 (+ 400us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:01.817937 (+ 29us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:01.826213 (+ 8276us) server_negotiation.cc:373] Connection header received 0627 19:32:01.826424 (+ 211us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:01.826477 (+ 53us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:01.826606 (+ 129us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:01.826735 (+ 129us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:01.851748 (+ 25013us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.852150 (+ 402us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.857848 (+ 5698us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:01.859001 (+ 1153us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:01.859194 (+ 193us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:01.872697 (+ 13503us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:01.872725 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:01.872750 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:01.872838 (+ 88us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:01.879601 (+ 6763us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.889553 (+ 9952us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.889581 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.889600 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.890278 (+ 678us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:01.896287 (+ 6009us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:01.896320 (+ 33us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:01.896339 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:01.896865 (+ 526us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:01.897182 (+ 317us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:01.899702 (+ 2520us) server_negotiation.cc:264] Negotiation successful 0627 19:32:01.900255 (+ 553us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":156} I0627 19:32:01.963454 23529 raft_consensus.cc:1081] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [term 1 FOLLOWER]: Refusing update from remote peer b9d10270bf864c51bdcf2f696e252fdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0627 19:32:01.965108 24322 consensus_queue.cc:962] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0627 19:32:01.975687 23359 raft_consensus.cc:1081] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [term 1 FOLLOWER]: Refusing update from remote peer b9d10270bf864c51bdcf2f696e252fdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0627 19:32:01.977030 24322 consensus_queue.cc:962] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0627 19:32:02.030138 23574 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:02.030356 23574 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:32:02.030882 24333 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9d10270bf864c51bdcf2f696e252fdd. Latest consensus state: current_term: 1 leader_uuid: "b9d10270bf864c51bdcf2f696e252fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:32:02.032016 24333 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:02.034099 24334 sys_catalog.cc:337] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9d10270bf864c51bdcf2f696e252fdd. Latest consensus state: current_term: 1 leader_uuid: "b9d10270bf864c51bdcf2f696e252fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:32:02.035406 24334 sys_catalog.cc:340] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:02.058033 24320 sys_catalog.cc:337] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9d10270bf864c51bdcf2f696e252fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:32:02.059296 24334 sys_catalog.cc:337] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9d10270bf864c51bdcf2f696e252fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:32:02.060034 24334 sys_catalog.cc:340] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50 [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:02.061159 24320 sys_catalog.cc:340] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [sys.catalog]: This master's current role is: LEADER I0627 19:32:02.063782 24329 catalog_manager.cc:904] Generated new certificate authority record I0627 19:32:02.066062 24329 catalog_manager.cc:1026] Loading token signing keys... I0627 19:32:02.065482 24340 sys_catalog.cc:337] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9d10270bf864c51bdcf2f696e252fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:32:02.066792 24340 sys_catalog.cc:340] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd [sys.catalog]: This master's current role is: LEADER I0627 19:32:02.097609 24329 catalog_manager.cc:4102] T 00000000000000000000000000000000 P b9d10270bf864c51bdcf2f696e252fdd: Generated new TSK 0 I0627 19:32:02.116677 24333 sys_catalog.cc:337] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9d10270bf864c51bdcf2f696e252fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d10270bf864c51bdcf2f696e252fdd" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 51131 } } peers { permanent_uuid: "c7c68ec568384c83885614bc22ec1a50" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 46323 } } peers { permanent_uuid: "6e88e61dad8a4be78044c758ab7bc93c" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 44480 } } } I0627 19:32:02.117308 24333 sys_catalog.cc:340] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:02.168442 23158 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.1@KRBTEST.COM'} at 127.12.131.1:35967 W0627 19:32:02.264780 24308 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:02.253174 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60594 0627 19:32:02.253560 (+ 386us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:02.253583 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:02.260530 (+ 6947us) server_negotiation.cc:373] Connection header received 0627 19:32:02.261295 (+ 765us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:02.261323 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:02.261489 (+ 166us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:02.261631 (+ 142us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:02.263949 (+ 2318us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:60594: BlockingRecv error: recv got EOF from 127.12.131.61:60594 (error 108) Metrics: {"server-negotiator.queue_time_us":145} W0627 19:32:02.272636 24319 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:02.254112 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:47990 0627 19:32:02.254467 (+ 355us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:02.254490 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:02.266334 (+ 11844us) server_negotiation.cc:373] Connection header received 0627 19:32:02.266758 (+ 424us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:02.266782 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:02.266908 (+ 126us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:02.267043 (+ 135us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:02.271882 (+ 4839us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.62:47990: BlockingRecv error: recv got EOF from 127.12.131.62:47990 (error 108) Metrics: {"server-negotiator.queue_time_us":118} W0627 19:32:02.278407 24306 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:02.273870 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:37945 0627 19:32:02.274166 (+ 296us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:02.274190 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:02.275031 (+ 841us) server_negotiation.cc:373] Connection header received 0627 19:32:02.275280 (+ 249us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:02.275307 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:02.275466 (+ 159us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:02.275610 (+ 144us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:02.277615 (+ 2005us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.60:37945: BlockingRecv error: recv got EOF from 127.12.131.60:37945 (error 108) Metrics: {"server-negotiator.queue_time_us":83} W0627 19:32:02.321949 24308 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:02.310032 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60597 0627 19:32:02.310369 (+ 337us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:02.310399 (+ 30us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:02.318305 (+ 7906us) server_negotiation.cc:373] Connection header received 0627 19:32:02.318901 (+ 596us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:02.318918 (+ 17us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:02.319055 (+ 137us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:02.319178 (+ 123us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:02.321151 (+ 1973us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:60597: BlockingRecv error: recv got EOF from 127.12.131.61:60597 (error 108) Metrics: {"server-negotiator.queue_time_us":91} W0627 19:32:02.364068 24308 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:02.351521 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60598 0627 19:32:02.351830 (+ 309us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:02.351853 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:02.360116 (+ 8263us) server_negotiation.cc:373] Connection header received 0627 19:32:02.360619 (+ 503us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:02.360647 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:02.360813 (+ 166us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:02.360951 (+ 138us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:02.363262 (+ 2311us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:60598: BlockingRecv error: recv got EOF from 127.12.131.61:60598 (error 108) Metrics: {"server-negotiator.queue_time_us":98} W0627 19:32:02.408427 24308 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:02.395641 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60599 0627 19:32:02.396010 (+ 369us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:02.396034 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:02.404332 (+ 8298us) server_negotiation.cc:373] Connection header received 0627 19:32:02.405112 (+ 780us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:02.405140 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:02.405313 (+ 173us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:02.405480 (+ 167us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:02.407600 (+ 2120us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:60599: BlockingRecv error: recv got EOF from 127.12.131.61:60599 (error 108) Metrics: {"server-negotiator.queue_time_us":143} W0627 19:32:02.463660 24308 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:02.450738 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60600 0627 19:32:02.451066 (+ 328us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:02.451087 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:02.458449 (+ 7362us) server_negotiation.cc:373] Connection header received 0627 19:32:02.458740 (+ 291us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:02.458766 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:02.458929 (+ 163us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:02.459060 (+ 131us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:02.462854 (+ 3794us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:60600: BlockingRecv error: recv got EOF from 127.12.131.61:60600 (error 108) Metrics: {"server-negotiator.queue_time_us":134} W0627 19:32:02.523705 24308 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:02.515792 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60601 0627 19:32:02.516140 (+ 348us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:02.516163 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:02.519195 (+ 3032us) server_negotiation.cc:373] Connection header received 0627 19:32:02.519337 (+ 142us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:02.519389 (+ 52us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:02.519553 (+ 164us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:02.519684 (+ 131us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:02.522903 (+ 3219us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:60601: BlockingRecv error: recv got EOF from 127.12.131.61:60601 (error 108) Metrics: {"server-negotiator.queue_time_us":137} W0627 19:32:02.601801 24308 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:02.594280 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60602 0627 19:32:02.594842 (+ 562us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:02.594864 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:02.596290 (+ 1426us) server_negotiation.cc:373] Connection header received 0627 19:32:02.596769 (+ 479us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:02.596798 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:02.596956 (+ 158us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:02.597095 (+ 139us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:02.601215 (+ 4120us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:60602: BlockingRecv error: recv got EOF from 127.12.131.61:60602 (error 108) Metrics: {"server-negotiator.queue_time_us":353} W0627 19:32:02.663748 24308 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:02.657709 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60603 0627 19:32:02.658003 (+ 294us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:02.658026 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:02.659977 (+ 1951us) server_negotiation.cc:373] Connection header received 0627 19:32:02.660811 (+ 834us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:02.660835 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:02.660981 (+ 146us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:02.661107 (+ 126us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:02.663027 (+ 1920us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:60603: BlockingRecv error: recv got EOF from 127.12.131.61:60603 (error 108) Metrics: {"server-negotiator.queue_time_us":90} I0627 19:32:02.749187 23158 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.3@KRBTEST.COM'} at 127.12.131.3:33914 W0627 19:32:02.781636 24308 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:02.776692 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60604 0627 19:32:02.777036 (+ 344us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:02.777058 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:02.778105 (+ 1047us) server_negotiation.cc:373] Connection header received 0627 19:32:02.778567 (+ 462us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:02.778592 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:02.778737 (+ 145us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:02.778892 (+ 155us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:02.780866 (+ 1974us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:60604: BlockingRecv error: recv got EOF from 127.12.131.61:60604 (error 108) Metrics: {"server-negotiator.queue_time_us":143} I0627 19:32:02.849580 23586 catalog_manager.cc:1062] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: acquiring CA information for follower catalog manager: success I0627 19:32:02.852926 23586 catalog_manager.cc:1090] T 00000000000000000000000000000000 P c7c68ec568384c83885614bc22ec1a50: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0627 19:32:02.858680 23158 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.2@KRBTEST.COM'} at 127.12.131.2:35421 I0627 19:32:02.940582 24308 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:02.927385 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60605 0627 19:32:02.927730 (+ 345us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:02.927756 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:02.929839 (+ 2083us) server_negotiation.cc:373] Connection header received 0627 19:32:02.930225 (+ 386us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:02.930250 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:02.930432 (+ 182us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:02.930566 (+ 134us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:32:02.931686 (+ 1120us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:02.932110 (+ 424us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:02.935732 (+ 3622us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:02.936800 (+ 1068us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:02.936977 (+ 177us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:02.938340 (+ 1363us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:32:02.938770 (+ 430us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:32:02.938882 (+ 112us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:02.939258 (+ 376us) server_negotiation.cc:264] Negotiation successful 0627 19:32:02.939446 (+ 188us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":106} W0627 19:32:03.016441 24406 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:02.991580 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:37955 0627 19:32:03.007039 (+ 15459us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:03.007062 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:03.007098 (+ 36us) server_negotiation.cc:373] Connection header received 0627 19:32:03.007244 (+ 146us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:03.007269 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:03.007468 (+ 199us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:03.007610 (+ 142us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:03.015633 (+ 8023us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.60:37955: BlockingRecv error: recv got EOF from 127.12.131.60:37955 (error 108) Metrics: {"server-negotiator.queue_time_us":15261,"thread_start_us":16163,"threads_started":1} I0627 19:32:03.036558 23574 catalog_manager.cc:1062] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: acquiring CA information for follower catalog manager: success I0627 19:32:03.039503 23574 catalog_manager.cc:1090] T 00000000000000000000000000000000 P 6e88e61dad8a4be78044c758ab7bc93c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0627 19:32:03.174135 23777 heartbeater.cc:467] Master 127.12.131.62:51131 was elected leader, sending a full tablet report... I0627 19:32:03.297025 24308 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:03.287968 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60607 0627 19:32:03.288257 (+ 289us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:03.288279 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:03.289474 (+ 1195us) server_negotiation.cc:373] Connection header received 0627 19:32:03.289777 (+ 303us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:03.289802 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:03.289950 (+ 148us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:03.290072 (+ 122us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:32:03.291200 (+ 1128us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:03.291645 (+ 445us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:03.293239 (+ 1594us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:03.294260 (+ 1021us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:03.294433 (+ 173us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:03.295099 (+ 666us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:32:03.295295 (+ 196us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:32:03.295396 (+ 101us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:03.295674 (+ 278us) server_negotiation.cc:264] Negotiation successful 0627 19:32:03.295827 (+ 153us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":84} I0627 19:32:03.325887 24406 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:03.316239 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:37957 0627 19:32:03.316592 (+ 353us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:03.316615 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:03.317791 (+ 1176us) server_negotiation.cc:373] Connection header received 0627 19:32:03.318075 (+ 284us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:03.318102 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:03.318268 (+ 166us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:03.318430 (+ 162us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:32:03.319132 (+ 702us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:03.319623 (+ 491us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:03.321249 (+ 1626us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:03.322288 (+ 1039us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:03.322495 (+ 207us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:03.323375 (+ 880us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:32:03.323786 (+ 411us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:32:03.323910 (+ 124us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:03.324174 (+ 264us) server_negotiation.cc:264] Negotiation successful 0627 19:32:03.324736 (+ 562us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":133} I0627 19:32:03.377696 24308 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:03.350792 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:60610 0627 19:32:03.351076 (+ 284us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:03.351098 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:03.363464 (+ 12366us) server_negotiation.cc:373] Connection header received 0627 19:32:03.363716 (+ 252us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:03.363742 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:03.363896 (+ 154us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:03.364000 (+ 104us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:32:03.364759 (+ 759us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:03.365023 (+ 264us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:03.373179 (+ 8156us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:03.374223 (+ 1044us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:03.374431 (+ 208us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:03.375603 (+ 1172us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:32:03.375804 (+ 201us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:32:03.375894 (+ 90us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:03.376381 (+ 487us) server_negotiation.cc:264] Negotiation successful 0627 19:32:03.376540 (+ 159us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":85} I0627 19:32:03.385375 24422 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:03.350373 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:48004 0627 19:32:03.361833 (+ 11460us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:03.361861 (+ 28us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:03.372243 (+ 10382us) server_negotiation.cc:373] Connection header received 0627 19:32:03.372598 (+ 355us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:03.372624 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:03.372802 (+ 178us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:03.372946 (+ 144us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:32:03.374188 (+ 1242us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:03.375550 (+ 1362us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:03.380317 (+ 4767us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:03.381435 (+ 1118us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:03.381632 (+ 197us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:03.382486 (+ 854us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:32:03.382868 (+ 382us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:32:03.383716 (+ 848us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:03.383826 (+ 110us) server_negotiation.cc:264] Negotiation successful 0627 19:32:03.383995 (+ 169us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":9611,"thread_start_us":9342,"threads_started":1} I0627 19:32:03.387635 24406 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:03.372140 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:37960 0627 19:32:03.372475 (+ 335us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:03.372497 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:03.379383 (+ 6886us) server_negotiation.cc:373] Connection header received 0627 19:32:03.379781 (+ 398us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:03.379806 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:03.379979 (+ 173us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:03.380092 (+ 113us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:32:03.381418 (+ 1326us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:03.381822 (+ 404us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:03.383487 (+ 1665us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:03.384515 (+ 1028us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:03.384677 (+ 162us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:03.385657 (+ 980us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:32:03.385843 (+ 186us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:32:03.385929 (+ 86us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:03.386281 (+ 352us) server_negotiation.cc:264] Negotiation successful 0627 19:32:03.386464 (+ 183us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":103} I0627 19:32:03.390525 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 23563 W0627 19:32:03.419155 23467 connection.cc:512] server connection from 127.12.131.1:59010 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:59010): Connection reset by peer (error 104) W0627 19:32:03.419215 23129 connection.cc:512] server connection from 127.12.131.1:35967 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:35967): Connection reset by peer (error 104) I0627 19:32:03.420217 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 23797 W0627 19:32:03.422958 23278 connection.cc:512] server connection from 127.12.131.1:57252 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:57252): Connection reset by peer (error 104) W0627 19:32:03.465333 23278 connection.cc:512] server connection from 127.12.131.2:52669 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:52669): Connection reset by peer (error 104) W0627 19:32:03.465412 23136 connection.cc:512] server connection from 127.12.131.2:35421 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:35421): Connection reset by peer (error 104) W0627 19:32:03.466635 23468 connection.cc:512] server connection from 127.12.131.2:33354 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:33354): Connection reset by peer (error 104) I0627 19:32:03.467515 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 24056 W0627 19:32:03.499269 23126 connection.cc:512] server connection from 127.12.131.3:33914 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.3:33914): Connection reset by peer (error 104) I0627 19:32:03.500213 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 23061 W0627 19:32:03.500213 23278 connection.cc:512] server connection from 127.12.131.3:60338 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.3:60338): Connection reset by peer (error 104) W0627 19:32:03.500977 23487 connection.cc:512] server connection from 127.12.131.3:42574 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.3:42574): Connection reset by peer (error 104) I0627 19:32:03.536069 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 23223 I0627 19:32:03.571650 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 23402 [ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (11925 ms) [ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 Loading random data Initializing database '/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/principal' for realm 'KRBTEST.COM', master key name 'K/M@KRBTEST.COM' Jun 27 19:32:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): setting up network... Jun 27 19:32:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): listening on fd 7: udp 0.0.0.0.0 (pktinfo) krb5kdc: setsockopt(8,IPV6_V6ONLY,1) worked krb5kdc: Invalid argument - Cannot request packet info for udp socket address :: port 0 Jun 27 19:32:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): skipping unrecognized local address family 17 Jun 27 19:32:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): skipping unrecognized local address family 17 Jun 27 19:32:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): set up 1 sockets Jun 27 19:32:03 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): commencing operation krb5kdc: starting... W0627 19:32:03.767761 12812 mini_kdc.cc:117] Time spent starting KDC: real 0.137s user 0.002s sys 0.015s WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-admin@KRBTEST.COM" created. W0627 19:32:03.918362 12812 mini_kdc.cc:250] Time spent creating user principal test-admin: real 0.150s user 0.000s sys 0.006s WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "test-user@KRBTEST.COM" created. WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy Authenticating as principal jenkins/admin@KRBTEST.COM with password. Principal "joe-interloper@KRBTEST.COM" created. Jun 27 19:32:04 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM Password for test-admin@KRBTEST.COM: WARNING: no policy specified for kudu/127.12.131.62@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.62@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. Entry for principal kudu/127.12.131.62 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab. W0627 19:32:04.268932 12812 mini_kdc.cc:260] Time spent creating service keytab for kudu/127.12.131.62: real 0.209s user 0.000s sys 0.012s I0627 19:32:04.270318 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:54619 --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --enable_leader_failure_detection=false --rpc_authentication=required --rpc_encryption=required --rpc_reuseport=true --master_addresses=127.12.131.62:54619,127.12.131.61:54937,127.12.131.60:47715 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:32:04.532204 24443 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0627 19:32:04.532954 24443 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:32:04.533152 24443 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.62 W0627 19:32:04.533576 24443 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:32:04.556536 24443 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:32:04.556843 24443 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:32:04.557070 24443 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:32:04.557343 24443 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:32:04.557556 24443 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:32:04.557765 24443 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.62 I0627 19:32:04.599542 24443 master_main.cc:73] Master server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --master_addresses=127.12.131.62:54619,127.12.131.61:54937,127.12.131.60:47715 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.62:54619 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.62.keytab --principal=kudu/127.12.131.62 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/data/info.pb --webserver_interface=127.12.131.62 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:32:04.614919 24443 master_main.cc:80] Initializing master server... I0627 19:32:04.618764 24443 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 231092us Jun 27 19:32:04 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153124, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:32:04.625258 24443 init.cc:339] Logged in from keytab as kudu/127.12.131.62@KRBTEST.COM (short username kudu) I0627 19:32:04.634872 24443 fs_manager.cc:261] Metadata directory not provided I0627 19:32:04.635108 24443 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:32:04.635545 24443 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:32:04.635746 24443 server_base.cc:433] Attempting to create new FS layout instead I0627 19:32:04.652801 24443 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/data/instance: uuid: "8dbb31411daf4b098e97fab30c7476ba" format_stamp: "Formatted at 2018-06-28 02:32:04 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:04.653908 24443 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/wal/instance: uuid: "8dbb31411daf4b098e97fab30c7476ba" format_stamp: "Formatted at 2018-06-28 02:32:04 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:04.679069 24443 fs_manager.cc:501] Time spent creating directory manager: real 0.025s user 0.014s sys 0.005s I0627 19:32:04.681048 24443 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:32:04.682550 24443 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:32:04.697026 24443 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s I0627 19:32:04.697315 24443 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/wal uuid: "8dbb31411daf4b098e97fab30c7476ba" format_stamp: "Formatted at 2018-06-28 02:32:04 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:04.697767 24443 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:32:04.985266 24443 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:32:04.993038 24443 master_main.cc:83] Starting Master server... I0627 19:32:05.203634 24443 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.62:54619 I0627 19:32:05.205209 24443 webserver.cc:175] Starting webserver on 127.12.131.62:0 I0627 19:32:05.205384 24443 webserver.cc:186] Document root disabled I0627 19:32:05.208010 24443 webserver.cc:313] Webserver started. Bound to: http://127.12.131.62:51099/ I0627 19:32:05.209074 24443 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-0/data/info.pb I0627 19:32:05.214375 24507 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:32:05.216055 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 24443 I0627 19:32:05.234341 24507 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } has no permanent_uuid. Determining permanent_uuid... WARNING: no policy specified for kudu/127.12.131.61@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.61@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.61.keytab. Entry for principal kudu/127.12.131.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.61.keytab. Entry for principal kudu/127.12.131.61 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.61.keytab. Entry for principal kudu/127.12.131.61 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.61.keytab. I0627 19:32:05.269944 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.61:54937 --webserver_interface=127.12.131.61 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.61.keytab --principal=kudu/127.12.131.61 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --enable_leader_failure_detection=false --rpc_authentication=required --rpc_encryption=required --rpc_reuseport=true --master_addresses=127.12.131.62:54619,127.12.131.61:54937,127.12.131.60:47715 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} Jun 27 19:32:05 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153124, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:32:05.296563 24509 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:05.238762 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:54619 0627 19:32:05.245410 (+ 6648us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:05.245523 (+ 113us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:05.263961 (+ 18438us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:05.265251 (+ 1290us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:05.265311 (+ 60us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:05.267235 (+ 1924us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:05.267540 (+ 305us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:05.267917 (+ 377us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:05.267969 (+ 52us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:05.268978 (+ 1009us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:05.269004 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:05.269904 (+ 900us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:05.269945 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:05.273213 (+ 3268us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:05.273241 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:05.273557 (+ 316us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:05.273756 (+ 199us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:05.273780 (+ 24us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:05.276849 (+ 3069us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:05.287481 (+ 10632us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:05.287511 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:05.287536 (+ 25us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:05.288787 (+ 1251us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:05.290702 (+ 1915us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:05.290731 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:05.290752 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:05.290965 (+ 213us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:05.292092 (+ 1127us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:05.292124 (+ 32us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:05.292515 (+ 391us) client_negotiation.cc:695] Sending connection context 0627 19:32:05.292802 (+ 287us) client_negotiation.cc:225] Negotiation successful 0627 19:32:05.293384 (+ 582us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":6290,"thread_start_us":5308,"threads_started":1} I0627 19:32:05.300860 24510 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:05.246708 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:36393 0627 19:32:05.263081 (+ 16373us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:05.263114 (+ 33us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:05.263186 (+ 72us) server_negotiation.cc:373] Connection header received 0627 19:32:05.264557 (+ 1371us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:05.264588 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:05.264851 (+ 263us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:05.265041 (+ 190us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:05.268198 (+ 3157us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:05.268742 (+ 544us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:05.270854 (+ 2112us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:05.272599 (+ 1745us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:05.272815 (+ 216us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:05.277915 (+ 5100us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:05.277954 (+ 39us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:05.277985 (+ 31us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:05.278063 (+ 78us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:05.286765 (+ 8702us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:05.289717 (+ 2952us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:05.289750 (+ 33us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:05.289769 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:05.290531 (+ 762us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:05.291187 (+ 656us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:05.291213 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:05.291232 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:05.291889 (+ 657us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:05.297563 (+ 5674us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:05.297829 (+ 266us) server_negotiation.cc:264] Negotiation successful 0627 19:32:05.298337 (+ 508us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":16127,"thread_start_us":16730,"threads_started":1} I0627 19:32:05.303293 24507 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } has no permanent_uuid. Determining permanent_uuid... I0627 19:32:05.305178 24509 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:05.304209 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:54937 0627 19:32:05.304575 (+ 366us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:05.304755 (+ 180us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.61:54937: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":106} W0627 19:32:05.306694 24507 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.61:54937: Network error: Client connection negotiation failed: client connection to 127.12.131.61:54937: connect: Connection refused (error 111) I0627 19:32:05.326133 24507 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } attempt: 1 I0627 19:32:05.328014 24509 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:05.326951 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:54937 0627 19:32:05.327558 (+ 607us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:05.327717 (+ 159us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.61:54937: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":114} W0627 19:32:05.328474 24507 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.61:54937: Network error: Client connection negotiation failed: client connection to 127.12.131.61:54937: connect: Connection refused (error 111) I0627 19:32:05.376843 24507 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } attempt: 2 I0627 19:32:05.378615 24509 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:05.377662 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:54937 0627 19:32:05.377971 (+ 309us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:05.378136 (+ 165us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.61:54937: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":96} W0627 19:32:05.379087 24507 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.61:54937: Network error: Client connection negotiation failed: client connection to 127.12.131.61:54937: connect: Connection refused (error 111) I0627 19:32:05.490453 24507 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } attempt: 3 I0627 19:32:05.492619 24509 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:05.491272 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:54937 0627 19:32:05.491606 (+ 334us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:05.492165 (+ 559us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.61:54937: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":105} W0627 19:32:05.493111 24507 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.61:54937: Network error: Client connection negotiation failed: client connection to 127.12.131.61:54937: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:32:05.540174 24512 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0627 19:32:05.541055 24512 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:32:05.541254 24512 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.61 W0627 19:32:05.541749 24512 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:32:05.561975 24512 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:32:05.562238 24512 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:32:05.562413 24512 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:32:05.562639 24512 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:32:05.562871 24512 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:32:05.563130 24512 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.61 I0627 19:32:05.607218 24512 master_main.cc:73] Master server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/wal --ipki_ca_key_size=1024 --master_addresses=127.12.131.62:54619,127.12.131.61:54937,127.12.131.60:47715 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.61:54937 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.61.keytab --principal=kudu/127.12.131.61 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/data/info.pb --webserver_interface=127.12.131.61 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:32:05.616736 24512 master_main.cc:80] Initializing master server... I0627 19:32:05.619534 24512 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 231592us Jun 27 19:32:05 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153125, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:32:05.624959 24512 init.cc:339] Logged in from keytab as kudu/127.12.131.61@KRBTEST.COM (short username kudu) I0627 19:32:05.630275 24512 fs_manager.cc:261] Metadata directory not provided I0627 19:32:05.630446 24512 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:32:05.630733 24512 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:32:05.630859 24512 server_base.cc:433] Attempting to create new FS layout instead I0627 19:32:05.640365 24512 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/data/instance: uuid: "501dc3febd1c4dd191dac8eb0224c0c3" format_stamp: "Formatted at 2018-06-28 02:32:05 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:05.641219 24512 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/wal/instance: uuid: "501dc3febd1c4dd191dac8eb0224c0c3" format_stamp: "Formatted at 2018-06-28 02:32:05 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:05.655273 24512 fs_manager.cc:501] Time spent creating directory manager: real 0.014s user 0.007s sys 0.005s I0627 19:32:05.657204 24512 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:32:05.658670 24512 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:32:05.669486 24507 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } attempt: 4 I0627 19:32:05.671145 24509 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:05.670252 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:54937 0627 19:32:05.670567 (+ 315us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:05.670720 (+ 153us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.61:54937: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":105} W0627 19:32:05.671519 24507 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.61:54937: Network error: Client connection negotiation failed: client connection to 127.12.131.61:54937: connect: Connection refused (error 111) I0627 19:32:05.680728 24512 fs_manager.cc:417] Time spent opening block manager: real 0.010s user 0.001s sys 0.004s I0627 19:32:05.680927 24512 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/wal uuid: "501dc3febd1c4dd191dac8eb0224c0c3" format_stamp: "Formatted at 2018-06-28 02:32:05 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:05.681285 24512 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:32:05.944298 24512 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:32:05.952250 24512 master_main.cc:83] Starting Master server... I0627 19:32:05.974869 24507 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } attempt: 5 I0627 19:32:05.976565 24509 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:05.975620 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:54937 0627 19:32:05.975934 (+ 314us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:05.976086 (+ 152us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.61:54937: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":114} W0627 19:32:05.976930 24507 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.61:54937: Network error: Client connection negotiation failed: client connection to 127.12.131.61:54937: connect: Connection refused (error 111) I0627 19:32:06.151135 24512 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.61:54937 I0627 19:32:06.152568 24512 webserver.cc:175] Starting webserver on 127.12.131.61:0 I0627 19:32:06.152742 24512 webserver.cc:186] Document root disabled I0627 19:32:06.160352 24512 webserver.cc:313] Webserver started. Bound to: http://127.12.131.61:39062/ I0627 19:32:06.161407 24512 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-1/data/info.pb I0627 19:32:06.164978 24577 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0627 19:32:06.170969 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 24512 WARNING: no policy specified for kudu/127.12.131.60@KRBTEST.COM; defaulting to no policy I0627 19:32:06.183688 24577 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } has no permanent_uuid. Determining permanent_uuid... Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.60@KRBTEST.COM" created. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.60.keytab. Entry for principal kudu/127.12.131.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.60.keytab. Entry for principal kudu/127.12.131.60 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.60.keytab. Entry for principal kudu/127.12.131.60 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.60.keytab. Jun 27 19:32:06 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153125, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.61@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:32:06.214393 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.60:47715 --webserver_interface=127.12.131.60 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.60.keytab --principal=kudu/127.12.131.60 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --enable_leader_failure_detection=false --rpc_authentication=required --rpc_encryption=required --rpc_reuseport=true --master_addresses=127.12.131.62:54619,127.12.131.61:54937,127.12.131.60:47715 with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:32:06.230240 24579 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.187660 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:54619 0627 19:32:06.195785 (+ 8125us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:06.195885 (+ 100us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:06.200208 (+ 4323us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:06.206214 (+ 6006us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:06.206247 (+ 33us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:06.206935 (+ 688us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:06.207219 (+ 284us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:06.207529 (+ 310us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:06.207577 (+ 48us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:06.208448 (+ 871us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:06.208475 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:06.209482 (+ 1007us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:06.209525 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:06.211129 (+ 1604us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:06.211157 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:06.211479 (+ 322us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:06.211686 (+ 207us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:06.211707 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:06.214784 (+ 3077us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:06.222806 (+ 8022us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:06.222838 (+ 32us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:06.222866 (+ 28us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:06.224197 (+ 1331us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:06.225321 (+ 1124us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:06.225346 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:06.225366 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:06.225599 (+ 233us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:06.226554 (+ 955us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:06.226582 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:06.226996 (+ 414us) client_negotiation.cc:695] Sending connection context 0627 19:32:06.227291 (+ 295us) client_negotiation.cc:225] Negotiation successful 0627 19:32:06.227859 (+ 568us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":7725,"thread_start_us":7394,"threads_started":1} I0627 19:32:06.233283 24581 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.192708 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:36400 0627 19:32:06.205514 (+ 12806us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:06.205540 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:06.205581 (+ 41us) server_negotiation.cc:373] Connection header received 0627 19:32:06.205687 (+ 106us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:06.205711 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:06.205829 (+ 118us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:06.205974 (+ 145us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:06.207808 (+ 1834us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:06.208211 (+ 403us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:06.209788 (+ 1577us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:06.210883 (+ 1095us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:06.215085 (+ 4202us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:06.215447 (+ 362us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:06.215475 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:06.215497 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:06.215567 (+ 70us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:06.222563 (+ 6996us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:06.224486 (+ 1923us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:06.224512 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:06.224531 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:06.225150 (+ 619us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:06.225832 (+ 682us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:06.225862 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:06.225881 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:06.226334 (+ 453us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:06.230785 (+ 4451us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:06.231002 (+ 217us) server_negotiation.cc:264] Negotiation successful 0627 19:32:06.231442 (+ 440us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":12563,"thread_start_us":13385,"threads_started":1} I0627 19:32:06.235011 24577 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } has no permanent_uuid. Determining permanent_uuid... Jun 27 19:32:06 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153125, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.61@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM I0627 19:32:06.284415 24579 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.236443 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:54937 0627 19:32:06.236761 (+ 318us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:06.236797 (+ 36us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:06.236938 (+ 141us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:06.249989 (+ 13051us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:06.250019 (+ 30us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:06.250697 (+ 678us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:06.250847 (+ 150us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:06.251023 (+ 176us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:06.251066 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:06.253441 (+ 2375us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:06.253468 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:06.254250 (+ 782us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:06.254289 (+ 39us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:06.258478 (+ 4189us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:06.258507 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:06.258789 (+ 282us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:06.258943 (+ 154us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:06.258959 (+ 16us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:06.261603 (+ 2644us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:06.275372 (+ 13769us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:06.275433 (+ 61us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:06.275454 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:06.276701 (+ 1247us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:06.277808 (+ 1107us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:06.277839 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:06.277859 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:06.278009 (+ 150us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:06.281014 (+ 3005us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:06.281040 (+ 26us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:06.281334 (+ 294us) client_negotiation.cc:695] Sending connection context 0627 19:32:06.281582 (+ 248us) client_negotiation.cc:225] Negotiation successful 0627 19:32:06.282084 (+ 502us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":103} I0627 19:32:06.287792 24583 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.237182 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:45201 0627 19:32:06.249288 (+ 12106us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:06.249319 (+ 31us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:06.249399 (+ 80us) server_negotiation.cc:373] Connection header received 0627 19:32:06.249528 (+ 129us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:06.249556 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:06.249726 (+ 170us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:06.249890 (+ 164us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:06.252523 (+ 2633us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:06.253249 (+ 726us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:06.256223 (+ 2974us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:06.258244 (+ 2021us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:06.258459 (+ 215us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:06.261873 (+ 3414us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:06.261913 (+ 40us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:06.261943 (+ 30us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:06.262020 (+ 77us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:06.275169 (+ 13149us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:06.276887 (+ 1718us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:06.276917 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:06.276936 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:06.277649 (+ 713us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:06.278165 (+ 516us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:06.278192 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:06.278212 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:06.280599 (+ 2387us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:06.281063 (+ 464us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:06.281843 (+ 780us) server_negotiation.cc:264] Negotiation successful 0627 19:32:06.282431 (+ 588us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":11871,"thread_start_us":12857,"threads_started":1} I0627 19:32:06.289993 24577 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } has no permanent_uuid. Determining permanent_uuid... I0627 19:32:06.291851 24579 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.290851 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:06.291249 (+ 398us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:06.291430 (+ 181us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":215} W0627 19:32:06.293824 24577 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:47715: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) I0627 19:32:06.313263 24577 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } attempt: 1 I0627 19:32:06.315240 24579 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.314031 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:06.314315 (+ 284us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:06.314492 (+ 177us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":84} W0627 19:32:06.315646 24577 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:47715: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) I0627 19:32:06.364017 24577 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } attempt: 2 I0627 19:32:06.365571 24579 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.364850 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:06.365142 (+ 292us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:06.365258 (+ 116us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":83} W0627 19:32:06.365967 24577 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:47715: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:32:06.444876 24582 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0627 19:32:06.445652 24582 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:32:06.445857 24582 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.60 W0627 19:32:06.446281 24582 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:32:06.465440 24582 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0627 19:32:06.465749 24582 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:32:06.465947 24582 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:32:06.466477 24582 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0627 19:32:06.466715 24582 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true W0627 19:32:06.466953 24582 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.60 I0627 19:32:06.477321 24577 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } attempt: 3 I0627 19:32:06.479223 24579 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.478150 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:06.478618 (+ 468us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:06.478775 (+ 157us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":263} W0627 19:32:06.479805 24577 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:47715: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) I0627 19:32:06.504218 24582 master_main.cc:73] Master server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/wal --ipki_ca_key_size=1024 --master_addresses=127.12.131.62:54619,127.12.131.61:54937,127.12.131.60:47715 --rpc_trace_negotiation=true --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.12.131.60:47715 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.60.keytab --principal=kudu/127.12.131.60 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/data/info.pb --webserver_interface=127.12.131.60 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:32:06.519469 24582 master_main.cc:80] Initializing master server... I0627 19:32:06.521793 24582 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 232092us Jun 27 19:32:06 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153126, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:32:06.524963 24582 init.cc:339] Logged in from keytab as kudu/127.12.131.60@KRBTEST.COM (short username kudu) I0627 19:32:06.535008 24582 fs_manager.cc:261] Metadata directory not provided I0627 19:32:06.535215 24582 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:32:06.535616 24582 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:32:06.535787 24582 server_base.cc:433] Attempting to create new FS layout instead I0627 19:32:06.560714 24582 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/data/instance: uuid: "aa795690fcf84c9890e0db9a92b11099" format_stamp: "Formatted at 2018-06-28 02:32:06 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:06.561836 24582 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/wal/instance: uuid: "aa795690fcf84c9890e0db9a92b11099" format_stamp: "Formatted at 2018-06-28 02:32:06 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:06.600709 24582 fs_manager.cc:501] Time spent creating directory manager: real 0.038s user 0.013s sys 0.003s I0627 19:32:06.602566 24582 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:32:06.603184 24582 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:32:06.628499 24582 fs_manager.cc:417] Time spent opening block manager: real 0.010s user 0.001s sys 0.001s I0627 19:32:06.628741 24582 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/wal uuid: "aa795690fcf84c9890e0db9a92b11099" format_stamp: "Formatted at 2018-06-28 02:32:06 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:06.629112 24582 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:32:06.511284 24507 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } attempt: 6 Jun 27 19:32:06 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153124, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM I0627 19:32:06.656221 24577 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } attempt: 4 I0627 19:32:06.657807 24579 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.656923 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:06.657195 (+ 272us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:06.657339 (+ 144us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":73} W0627 19:32:06.658751 24577 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:47715: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) I0627 19:32:06.676337 24595 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.636622 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:54937 0627 19:32:06.648820 (+ 12198us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:06.648864 (+ 44us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:06.649007 (+ 143us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:06.649590 (+ 583us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:06.649615 (+ 25us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:06.650273 (+ 658us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:06.650479 (+ 206us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:06.650644 (+ 165us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:06.650684 (+ 40us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:06.651454 (+ 770us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:06.651481 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:06.652281 (+ 800us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:06.652324 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:06.653756 (+ 1432us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:06.653782 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:06.654031 (+ 249us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:06.654186 (+ 155us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:06.654204 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:06.656936 (+ 2732us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:06.663950 (+ 7014us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:06.663978 (+ 28us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:06.663999 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:06.665178 (+ 1179us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:06.671993 (+ 6815us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:06.672022 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:06.672043 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:06.672223 (+ 180us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:06.673324 (+ 1101us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:06.673348 (+ 24us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:06.673679 (+ 331us) client_negotiation.cc:695] Sending connection context 0627 19:32:06.673888 (+ 209us) client_negotiation.cc:225] Negotiation successful 0627 19:32:06.674407 (+ 519us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":11984,"thread_start_us":11686,"threads_started":1} I0627 19:32:06.676790 24583 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.636815 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:45206 0627 19:32:06.637146 (+ 331us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:06.637172 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:06.648980 (+ 11808us) server_negotiation.cc:373] Connection header received 0627 19:32:06.649200 (+ 220us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:06.649226 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:06.649357 (+ 131us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:06.649530 (+ 173us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:06.650875 (+ 1345us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:06.651262 (+ 387us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:06.652503 (+ 1241us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:06.653550 (+ 1047us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:06.653743 (+ 193us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:06.657315 (+ 3572us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:06.657366 (+ 51us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:06.657395 (+ 29us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:06.657468 (+ 73us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:06.663762 (+ 6294us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:06.665338 (+ 1576us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:06.665391 (+ 53us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:06.665410 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:06.671779 (+ 6369us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:06.672622 (+ 843us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:06.672649 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:06.672668 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:06.673161 (+ 493us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:06.673380 (+ 219us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:06.674062 (+ 682us) server_negotiation.cc:264] Negotiation successful 0627 19:32:06.674525 (+ 463us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":134} I0627 19:32:06.678092 24507 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } has no permanent_uuid. Determining permanent_uuid... I0627 19:32:06.680991 24595 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.680125 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:06.680418 (+ 293us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:06.680564 (+ 146us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":68} W0627 19:32:06.681399 24507 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:47715: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) I0627 19:32:06.713776 24507 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } attempt: 1 I0627 19:32:06.715507 24595 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.714599 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:06.714897 (+ 298us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:06.715052 (+ 155us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":96} W0627 19:32:06.715963 24507 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:47715: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) I0627 19:32:06.784325 24507 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } attempt: 2 I0627 19:32:06.785991 24595 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.785088 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:06.785411 (+ 323us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:06.785563 (+ 152us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":92} W0627 19:32:06.786409 24507 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:47715: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) I0627 19:32:06.863046 24507 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } attempt: 3 I0627 19:32:06.864852 24595 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:06.863983 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:06.864256 (+ 273us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:06.864426 (+ 170us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":71} W0627 19:32:06.868741 24507 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:47715: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) I0627 19:32:06.883927 24582 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:32:06.887199 24582 master_main.cc:83] Starting Master server... I0627 19:32:07.006103 24507 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } attempt: 4 I0627 19:32:07.007781 24595 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:07.006900 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:07.007180 (+ 280us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:07.007339 (+ 159us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":78} W0627 19:32:07.008204 24507 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:47715: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) I0627 19:32:06.962117 24577 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } attempt: 5 I0627 19:32:07.104444 24579 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:07.103627 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:07.103912 (+ 285us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:07.104053 (+ 141us) negotiation.cc:304] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) Metrics: {"client-negotiator.queue_time_us":78} W0627 19:32:07.104754 24577 consensus_peers.cc:565] Error getting permanent uuid from config peer 127.12.131.60:47715: Network error: Client connection negotiation failed: client connection to 127.12.131.60:47715: connect: Connection refused (error 111) I0627 19:32:07.137511 24582 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.60:47715 I0627 19:32:07.139101 24582 webserver.cc:175] Starting webserver on 127.12.131.60:0 I0627 19:32:07.139256 24582 webserver.cc:186] Document root disabled I0627 19:32:07.145421 24582 webserver.cc:313] Webserver started. Bound to: http://127.12.131.60:45311/ I0627 19:32:07.146456 24582 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/master-2/data/info.pb I0627 19:32:07.153079 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master as pid 24582 I0627 19:32:07.156523 24651 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed WARNING: no policy specified for kudu/127.12.131.1@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.1@KRBTEST.COM" created. I0627 19:32:07.177254 24651 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } has no permanent_uuid. Determining permanent_uuid... Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. Entry for principal kudu/127.12.131.1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab. I0627 19:32:07.207687 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/data --block_manager=log --rpc_bind_addresses=127.12.131.1:0 --local_ip_for_outbound_sockets=127.12.131.1 --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:54619,127.12.131.61:54937,127.12.131.60:47715 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_authentication=required --rpc_encryption=required with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} Jun 27 19:32:07 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153126, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.60@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:32:07.228757 24654 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:07.189968 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:54619 0627 19:32:07.194113 (+ 4145us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:07.194208 (+ 95us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:07.202846 (+ 8638us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:07.204074 (+ 1228us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:07.204106 (+ 32us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:07.204992 (+ 886us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:07.205289 (+ 297us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.205731 (+ 442us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:07.205779 (+ 48us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.206671 (+ 892us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.206698 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.207707 (+ 1009us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:07.207748 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.209586 (+ 1838us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.209615 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.209936 (+ 321us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:07.210190 (+ 254us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:07.210210 (+ 20us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:07.213592 (+ 3382us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:07.220900 (+ 7308us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:07.220932 (+ 32us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:07.220961 (+ 29us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:07.222337 (+ 1376us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:07.223476 (+ 1139us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:07.223502 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:07.223522 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:07.223760 (+ 238us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:07.224804 (+ 1044us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:07.224833 (+ 29us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:07.225291 (+ 458us) client_negotiation.cc:695] Sending connection context 0627 19:32:07.225658 (+ 367us) client_negotiation.cc:225] Negotiation successful 0627 19:32:07.226226 (+ 568us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":3610,"thread_start_us":3245,"threads_started":1} I0627 19:32:07.231634 24655 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:07.191260 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:36414 0627 19:32:07.195660 (+ 4400us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:07.195685 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:07.195723 (+ 38us) server_negotiation.cc:373] Connection header received 0627 19:32:07.203288 (+ 7565us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:07.203317 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:07.203475 (+ 158us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:07.203611 (+ 136us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:07.206004 (+ 2393us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.206409 (+ 405us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.207959 (+ 1550us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.209045 (+ 1086us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.209219 (+ 174us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:07.214017 (+ 4798us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:07.214048 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:07.214076 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:07.214159 (+ 83us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:07.220598 (+ 6439us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:07.222582 (+ 1984us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:07.222613 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:07.222634 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:07.223258 (+ 624us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:07.223949 (+ 691us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:07.223976 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:07.223996 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:07.224496 (+ 500us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:07.224668 (+ 172us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:07.229434 (+ 4766us) server_negotiation.cc:264] Negotiation successful 0627 19:32:07.229862 (+ 428us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":4196,"thread_start_us":3964,"threads_started":1} I0627 19:32:07.233319 24651 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } has no permanent_uuid. Determining permanent_uuid... Jun 27 19:32:07 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153126, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.60@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM I0627 19:32:07.259127 24654 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:07.234276 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:54937 0627 19:32:07.234872 (+ 596us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:07.234907 (+ 35us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:07.235041 (+ 134us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:07.237284 (+ 2243us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:07.237314 (+ 30us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:07.237969 (+ 655us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:07.238129 (+ 160us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.238303 (+ 174us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:07.238345 (+ 42us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.239356 (+ 1011us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.239407 (+ 51us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.240209 (+ 802us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:07.240255 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.241882 (+ 1627us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.241907 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.242167 (+ 260us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:07.242331 (+ 164us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:07.242349 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:07.244964 (+ 2615us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:07.252309 (+ 7345us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:07.252339 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:07.252360 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:07.253607 (+ 1247us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:07.254764 (+ 1157us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:07.254791 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:07.254811 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:07.254967 (+ 156us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:07.256031 (+ 1064us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:07.256056 (+ 25us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:07.256362 (+ 306us) client_negotiation.cc:695] Sending connection context 0627 19:32:07.256601 (+ 239us) client_negotiation.cc:225] Negotiation successful 0627 19:32:07.257082 (+ 481us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":401} I0627 19:32:07.262770 24657 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:07.235048 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:45215 0627 19:32:07.236718 (+ 1670us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:07.236745 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:07.236783 (+ 38us) server_negotiation.cc:373] Connection header received 0627 19:32:07.236892 (+ 109us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:07.236917 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:07.237038 (+ 121us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:07.238519 (+ 1481us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:07.238777 (+ 258us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.239163 (+ 386us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.240580 (+ 1417us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.241666 (+ 1086us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.245230 (+ 3564us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:07.245580 (+ 350us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:07.245608 (+ 28us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:07.245630 (+ 22us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:07.245705 (+ 75us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:07.252069 (+ 6364us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:07.253867 (+ 1798us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:07.253894 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:07.253913 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:07.254567 (+ 654us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:07.255228 (+ 661us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:07.255257 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:07.255277 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:07.255844 (+ 567us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:07.260270 (+ 4426us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:07.260510 (+ 240us) server_negotiation.cc:264] Negotiation successful 0627 19:32:07.260942 (+ 432us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1464,"thread_start_us":1233,"threads_started":1} I0627 19:32:07.264372 24651 sys_catalog.cc:304] member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } has no permanent_uuid. Determining permanent_uuid... Jun 27 19:32:07 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153126, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.60@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM I0627 19:32:07.291520 24507 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } attempt: 5 I0627 19:32:07.316040 24654 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:07.265239 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:07.267015 (+ 1776us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:07.267049 (+ 34us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:07.267180 (+ 131us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:07.275792 (+ 8612us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:07.275845 (+ 53us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:07.276393 (+ 548us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:07.276519 (+ 126us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.276696 (+ 177us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:07.276737 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.277803 (+ 1066us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.277829 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.278650 (+ 821us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:07.278691 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.280986 (+ 2295us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.281013 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.281272 (+ 259us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:07.281460 (+ 188us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:07.281476 (+ 16us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:07.284143 (+ 2667us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:07.293419 (+ 9276us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:07.293448 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:07.293470 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:07.294731 (+ 1261us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:07.300628 (+ 5897us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:07.300657 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:07.300678 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:07.300844 (+ 166us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:07.302853 (+ 2009us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:07.302879 (+ 26us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:07.303165 (+ 286us) client_negotiation.cc:695] Sending connection context 0627 19:32:07.303369 (+ 204us) client_negotiation.cc:225] Negotiation successful 0627 19:32:07.303876 (+ 507us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":1569} I0627 19:32:07.320258 24658 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:07.265880 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:50993 0627 19:32:07.275203 (+ 9323us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:07.275224 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:07.275259 (+ 35us) server_negotiation.cc:373] Connection header received 0627 19:32:07.275345 (+ 86us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:07.275367 (+ 22us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:07.275551 (+ 184us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:07.275732 (+ 181us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:07.276942 (+ 1210us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.277607 (+ 665us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.278952 (+ 1345us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.280782 (+ 1830us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.284410 (+ 3628us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:07.284786 (+ 376us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:07.284822 (+ 36us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:07.284849 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:07.284919 (+ 70us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:07.292026 (+ 7107us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:07.298626 (+ 6600us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:07.298657 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:07.298677 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:07.299428 (+ 751us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:07.301150 (+ 1722us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:07.301177 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:07.301197 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:07.302066 (+ 869us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:07.316806 (+ 14740us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:07.317143 (+ 337us) server_negotiation.cc:264] Negotiation successful 0627 19:32:07.317671 (+ 528us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":9138,"thread_start_us":8803,"threads_started":1} Jun 27 19:32:07 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153124, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.62@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM I0627 19:32:07.328793 24651 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: Bootstrap starting. I0627 19:32:07.333451 24651 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: No blocks or log segments found. Creating new log. I0627 19:32:07.336138 24651 log.cc:525] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: Log is configured to *not* fsync() on all Append() calls I0627 19:32:07.357276 24595 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:07.292311 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:07.292778 (+ 467us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:07.292812 (+ 34us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:07.292941 (+ 129us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:07.316664 (+ 23723us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:07.316693 (+ 29us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:07.317379 (+ 686us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:07.317564 (+ 185us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.317731 (+ 167us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:07.317777 (+ 46us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.320537 (+ 2760us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.320564 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.321349 (+ 785us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:07.321390 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.324283 (+ 2893us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.324309 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.324606 (+ 297us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:07.324774 (+ 168us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:07.324794 (+ 20us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:07.327539 (+ 2745us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:07.350644 (+ 23105us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:07.350675 (+ 31us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:07.350696 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:07.351964 (+ 1268us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:07.352968 (+ 1004us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:07.352995 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:07.353019 (+ 24us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:07.353160 (+ 141us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:07.354234 (+ 1074us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:07.354260 (+ 26us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:07.354574 (+ 314us) client_negotiation.cc:695] Sending connection context 0627 19:32:07.354761 (+ 187us) client_negotiation.cc:225] Negotiation successful 0627 19:32:07.355273 (+ 512us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":274} I0627 19:32:07.358130 24659 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:07.292428 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:50994 0627 19:32:07.316047 (+ 23619us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:07.316077 (+ 30us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:07.316119 (+ 42us) server_negotiation.cc:373] Connection header received 0627 19:32:07.316235 (+ 116us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:07.316260 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:07.316389 (+ 129us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:07.316577 (+ 188us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:07.319164 (+ 2587us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.320328 (+ 1164us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.322763 (+ 2435us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.324073 (+ 1310us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.324269 (+ 196us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:07.329449 (+ 5180us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:07.329479 (+ 30us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:07.329513 (+ 34us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:07.329597 (+ 84us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:07.350395 (+ 20798us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:07.352135 (+ 1740us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:07.352168 (+ 33us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:07.352189 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:07.352809 (+ 620us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:07.353312 (+ 503us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:07.353340 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:07.353359 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:07.354051 (+ 692us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:07.354273 (+ 222us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:07.354973 (+ 700us) server_negotiation.cc:264] Negotiation successful 0627 19:32:07.355408 (+ 435us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":23255,"thread_start_us":22191,"threads_started":1} I0627 19:32:07.364641 24507 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: Bootstrap starting. I0627 19:32:07.367851 24507 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: No blocks or log segments found. Creating new log. I0627 19:32:07.378150 24507 log.cc:525] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: Log is configured to *not* fsync() on all Append() calls I0627 19:32:07.391935 24507 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: No bootstrap required, opened a new log I0627 19:32:07.427155 24507 raft_consensus.cc:325] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } I0627 19:32:07.428007 24507 raft_consensus.cc:628] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dbb31411daf4b098e97fab30c7476ba, State: Initialized, Role: FOLLOWER I0627 19:32:07.428614 24507 consensus_queue.cc:227] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } I0627 19:32:07.437479 24507 sys_catalog.cc:337] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:07.437665 24662 sys_catalog.cc:337] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:07.438354 24507 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:07.438454 24662 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [sys.catalog]: This master's current role is: FOLLOWER WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:32:07.439635 24656 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:32:07.440101 24656 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.1 I0627 19:32:07.440277 24507 sys_catalog.cc:424] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [sys.catalog]: configured and running, proceeding with master startup. W0627 19:32:07.440943 24656 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0627 19:32:07.449623 24443 master_main.cc:86] Master server successfully started. I0627 19:32:07.360555 24651 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: No bootstrap required, opened a new log W0627 19:32:07.471042 24663 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:07.471299 24663 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:07.473181 24656 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:32:07.473434 24656 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:32:07.473734 24656 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.1 W0627 19:32:07.474323 24656 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.1 I0627 19:32:07.475158 24651 raft_consensus.cc:325] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } I0627 19:32:07.475980 24651 raft_consensus.cc:628] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa795690fcf84c9890e0db9a92b11099, State: Initialized, Role: FOLLOWER I0627 19:32:07.476536 24651 consensus_queue.cc:227] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } I0627 19:32:07.483142 24666 sys_catalog.cc:337] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:07.484051 24651 sys_catalog.cc:337] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:07.484758 24651 sys_catalog.cc:340] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:07.486145 24651 sys_catalog.cc:424] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:32:07.487181 24666 sys_catalog.cc:340] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:07.494498 24582 master_main.cc:86] Master server successfully started. W0627 19:32:07.502852 24667 catalog_manager.cc:1064] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:07.503108 24667 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:07.511901 24463 tablet.cc:1678] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:32:07.540174 24656 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.1:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.1.keytab --principal=kudu/127.12.131.1 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/data/info.pb --webserver_interface=127.12.131.1 --webserver_port=0 --tserver_master_addrs=127.12.131.62:54619,127.12.131.61:54937,127.12.131.60:47715 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:32:07.561753 24656 tablet_server_main.cc:85] Initializing tablet server... I0627 19:32:07.572294 24656 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 232592us Jun 27 19:32:07 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153127, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:32:07.575642 24656 init.cc:339] Logged in from keytab as kudu/127.12.131.1@KRBTEST.COM (short username kudu) I0627 19:32:07.610427 24656 fs_manager.cc:261] Metadata directory not provided I0627 19:32:07.610651 24656 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:32:07.611022 24656 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:32:07.611201 24656 server_base.cc:433] Attempting to create new FS layout instead I0627 19:32:07.635000 24656 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/data/instance: uuid: "fbd50e06440b4131b656d2627053720a" format_stamp: "Formatted at 2018-06-28 02:32:07 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:07.639111 24577 consensus_peers.cc:575] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } attempt: 6 W0627 19:32:07.640877 24605 tablet.cc:1678] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:32:07.641345 24656 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/wal/instance: uuid: "fbd50e06440b4131b656d2627053720a" format_stamp: "Formatted at 2018-06-28 02:32:07 on quasar-ynjtoc-1.vpc.cloudera.com" Jun 27 19:32:07 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153125, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.61@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM I0627 19:32:07.684314 24656 fs_manager.cc:501] Time spent creating directory manager: real 0.042s user 0.015s sys 0.002s I0627 19:32:07.686203 24675 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:07.639953 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:07.650093 (+ 10140us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:07.650138 (+ 45us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:07.650417 (+ 279us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:07.651195 (+ 778us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:07.651220 (+ 25us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:07.651956 (+ 736us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:07.652128 (+ 172us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.652302 (+ 174us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:07.652343 (+ 41us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.653185 (+ 842us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.653210 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.654040 (+ 830us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:07.654087 (+ 47us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.657638 (+ 3551us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.657672 (+ 34us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:07.657963 (+ 291us) client_negotiation.cc:497] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:07.658132 (+ 169us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:07.658151 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:07.668531 (+ 10380us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:07.676084 (+ 7553us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:07.676114 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:07.676136 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:07.677427 (+ 1291us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:07.678546 (+ 1119us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:07.678573 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:07.678593 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:07.678759 (+ 166us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:07.679840 (+ 1081us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:07.679867 (+ 27us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:07.680174 (+ 307us) client_negotiation.cc:695] Sending connection context 0627 19:32:07.680420 (+ 246us) client_negotiation.cc:225] Negotiation successful 0627 19:32:07.680934 (+ 514us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":9908,"thread_start_us":15178,"threads_started":1} I0627 19:32:07.686300 24656 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:32:07.686628 24659 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:07.640174 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:50995 0627 19:32:07.640524 (+ 350us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:07.640544 (+ 20us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:07.650236 (+ 9692us) server_negotiation.cc:373] Connection header received 0627 19:32:07.650627 (+ 391us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:07.650653 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:07.650775 (+ 122us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:07.650904 (+ 129us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:07.652557 (+ 1653us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.652941 (+ 384us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.655740 (+ 2799us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:07.656748 (+ 1008us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:07.656984 (+ 236us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:07.668882 (+ 11898us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:07.668913 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:07.668937 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:07.669027 (+ 90us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:07.675664 (+ 6637us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:07.677642 (+ 1978us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:07.677672 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:07.677690 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:07.678316 (+ 626us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:07.678955 (+ 639us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:07.678982 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:07.679005 (+ 23us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:07.679531 (+ 526us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:07.679717 (+ 186us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:07.682000 (+ 2283us) server_negotiation.cc:264] Negotiation successful 0627 19:32:07.682482 (+ 482us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":104} I0627 19:32:07.687377 24656 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:32:07.699739 24577 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: Bootstrap starting. I0627 19:32:07.703508 24577 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: No blocks or log segments found. Creating new log. I0627 19:32:07.710803 24577 log.cc:525] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: Log is configured to *not* fsync() on all Append() calls I0627 19:32:07.728338 24577 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: No bootstrap required, opened a new log I0627 19:32:07.731211 24656 fs_manager.cc:417] Time spent opening block manager: real 0.018s user 0.001s sys 0.008s I0627 19:32:07.731397 24656 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/wal uuid: "fbd50e06440b4131b656d2627053720a" format_stamp: "Formatted at 2018-06-28 02:32:07 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:07.731745 24656 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0627 19:32:07.755401 24577 raft_consensus.cc:325] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } I0627 19:32:07.761890 24577 raft_consensus.cc:628] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501dc3febd1c4dd191dac8eb0224c0c3, State: Initialized, Role: FOLLOWER I0627 19:32:07.762507 24577 consensus_queue.cc:227] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } I0627 19:32:07.774763 24680 sys_catalog.cc:337] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:07.775645 24577 sys_catalog.cc:337] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:07.776367 24577 sys_catalog.cc:340] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:07.777775 24577 sys_catalog.cc:424] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [sys.catalog]: configured and running, proceeding with master startup. I0627 19:32:07.778625 24680 sys_catalog.cc:340] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:07.793885 24512 master_main.cc:86] Master server successfully started. W0627 19:32:07.824787 24682 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:07.825047 24682 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:07.961617 24533 tablet.cc:1678] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: Can't schedule compaction. Clean time has not been advanced past its initial value. I0627 19:32:08.238602 24656 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:32:08.252753 24656 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:32:08.267568 24656 tablet_server_main.cc:90] Starting tablet server... W0627 19:32:08.473775 24663 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:08.473987 24663 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:08.505661 24667 catalog_manager.cc:1064] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:08.505897 24667 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:08.832373 24682 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:08.832609 24682 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:32:09.435603 24656 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.1:36187 I0627 19:32:09.437294 24656 webserver.cc:175] Starting webserver on 127.12.131.1:0 I0627 19:32:09.437485 24656 webserver.cc:186] Document root disabled I0627 19:32:09.445890 24656 webserver.cc:313] Webserver started. Bound to: http://127.12.131.1:32846/ I0627 19:32:09.447022 24656 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-0/data/info.pb I0627 19:32:09.450191 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 24656 W0627 19:32:09.476436 24663 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:09.476653 24663 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry WARNING: no policy specified for kudu/127.12.131.2@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.2@KRBTEST.COM" created. W0627 19:32:09.508214 24667 catalog_manager.cc:1064] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:09.508422 24667 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:32:09.511986 24656 tablet_server_main.cc:93] Tablet server successfully started. Jun 27 19:32:09 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153127, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM Jun 27 19:32:09 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153127, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM Jun 27 19:32:09 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153127, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.1@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. Entry for principal kudu/127.12.131.2 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab. W0627 19:32:09.579795 12812 mini_kdc.cc:260] Time spent creating service keytab for kudu/127.12.131.2: real 0.129s user 0.000s sys 0.014s I0627 19:32:09.581303 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/data --block_manager=log --rpc_bind_addresses=127.12.131.2:0 --local_ip_for_outbound_sockets=127.12.131.2 --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:54619,127.12.131.61:54937,127.12.131.60:47715 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_authentication=required --rpc_encryption=required with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} I0627 19:32:09.591941 24832 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:09.453043 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:54619 0627 19:32:09.470177 (+ 17134us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:09.470290 (+ 113us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:09.537830 (+ 67540us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:09.538845 (+ 1015us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:09.538879 (+ 34us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:09.545518 (+ 6639us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:09.546465 (+ 947us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:09.547550 (+ 1085us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:09.547590 (+ 40us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:09.548711 (+ 1121us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:09.548740 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:09.549609 (+ 869us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:09.549794 (+ 185us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:09.553358 (+ 3564us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:09.553385 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:09.553766 (+ 381us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:09.554002 (+ 236us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:09.554021 (+ 19us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:09.561505 (+ 7484us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:09.573644 (+ 12139us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:09.573673 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:09.573700 (+ 27us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:09.575132 (+ 1432us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:09.577628 (+ 2496us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:09.577655 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:09.577674 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:09.578021 (+ 347us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:09.588252 (+ 10231us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:09.588282 (+ 30us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:09.588813 (+ 531us) client_negotiation.cc:695] Sending connection context 0627 19:32:09.589552 (+ 739us) client_negotiation.cc:225] Negotiation successful 0627 19:32:09.589936 (+ 384us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":16555,"mutex_wait_us":5336,"thread_start_us":18934,"threads_started":1} I0627 19:32:09.595746 24831 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:09.452404 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:36447 0627 19:32:09.456669 (+ 4265us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:09.456695 (+ 26us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:09.470830 (+ 14135us) server_negotiation.cc:373] Connection header received 0627 19:32:09.538239 (+ 67409us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:09.538269 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:09.538402 (+ 133us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:09.538606 (+ 204us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:09.547838 (+ 9232us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:09.548267 (+ 429us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:09.550061 (+ 1794us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:09.551138 (+ 1077us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:09.551316 (+ 178us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:09.562249 (+ 10933us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:09.562331 (+ 82us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:09.562357 (+ 26us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:09.562470 (+ 113us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:09.569420 (+ 6950us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:09.576188 (+ 6768us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:09.576216 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:09.576236 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:09.576893 (+ 657us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:09.578850 (+ 1957us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:09.578888 (+ 38us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:09.586996 (+ 8108us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:09.587540 (+ 544us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:09.587886 (+ 346us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:09.592760 (+ 4874us) server_negotiation.cc:264] Negotiation successful 0627 19:32:09.593118 (+ 358us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":2843,"thread_start_us":2162,"threads_started":1} I0627 19:32:09.597230 24838 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:09.466439 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:54937 0627 19:32:09.490930 (+ 24491us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:09.491104 (+ 174us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:09.538217 (+ 47113us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:09.539082 (+ 865us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:09.539115 (+ 33us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:09.540622 (+ 1507us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:09.541038 (+ 416us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:09.541469 (+ 431us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:09.541521 (+ 52us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:09.542432 (+ 911us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:09.542462 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:09.543567 (+ 1105us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:09.543615 (+ 48us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:09.545515 (+ 1900us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:09.545545 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:09.545919 (+ 374us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:09.547572 (+ 1653us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:09.547594 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:09.575583 (+ 27989us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:09.583151 (+ 7568us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:09.583176 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:09.583193 (+ 17us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:09.584277 (+ 1084us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:09.592154 (+ 7877us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:09.592176 (+ 22us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:09.592193 (+ 17us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:09.592348 (+ 155us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:09.594160 (+ 1812us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:09.594185 (+ 25us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:09.594496 (+ 311us) client_negotiation.cc:695] Sending connection context 0627 19:32:09.594825 (+ 329us) client_negotiation.cc:225] Negotiation successful 0627 19:32:09.595136 (+ 311us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":24184,"mutex_wait_us":2737,"thread_start_us":17792,"threads_started":1} I0627 19:32:09.599849 24844 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:09.491795 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:37476 0627 19:32:09.493963 (+ 2168us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:09.493990 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:09.513133 (+ 19143us) server_negotiation.cc:373] Connection header received 0627 19:32:09.538283 (+ 25150us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:09.538314 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:09.538463 (+ 149us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:09.538607 (+ 144us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:09.545559 (+ 6952us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:09.546002 (+ 443us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:09.553320 (+ 7318us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:09.554523 (+ 1203us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:09.554700 (+ 177us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:09.573101 (+ 18401us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:09.573198 (+ 97us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:09.573225 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:09.573316 (+ 91us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:09.580140 (+ 6824us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:09.587669 (+ 7529us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:09.587698 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:09.587716 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:09.588384 (+ 668us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:09.595675 (+ 7291us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:09.595702 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:09.595720 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:09.596176 (+ 456us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:09.596535 (+ 359us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:09.597645 (+ 1110us) server_negotiation.cc:264] Negotiation successful 0627 19:32:09.597992 (+ 347us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":1958,"thread_start_us":1676,"threads_started":1} I0627 19:32:09.602136 24837 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:09.466100 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.1:35921 0627 19:32:09.468794 (+ 2694us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:09.468816 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:09.492984 (+ 24168us) server_negotiation.cc:373] Connection header received 0627 19:32:09.538482 (+ 45498us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:09.538510 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:09.538623 (+ 113us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:09.538756 (+ 133us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:09.541739 (+ 2983us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:09.542125 (+ 386us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:09.543820 (+ 1695us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:09.544797 (+ 977us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:09.544961 (+ 164us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:09.576088 (+ 31127us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:09.576167 (+ 79us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:09.576191 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:09.576269 (+ 78us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:09.582609 (+ 6340us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:09.584771 (+ 2162us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:09.584795 (+ 24us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:09.584810 (+ 15us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:09.585360 (+ 550us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:09.592928 (+ 7568us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:09.592954 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:09.592972 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:09.593529 (+ 557us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:09.593850 (+ 321us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:09.598075 (+ 4225us) server_negotiation.cc:264] Negotiation successful 0627 19:32:09.598435 (+ 360us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":2491,"thread_start_us":2260,"threads_started":1} I0627 19:32:09.601917 24846 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:09.492742 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:09.512779 (+ 20037us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:09.513019 (+ 240us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:09.537830 (+ 24811us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:09.539418 (+ 1588us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:09.539446 (+ 28us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:09.544365 (+ 4919us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:09.544634 (+ 269us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:09.545278 (+ 644us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:09.545316 (+ 38us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:09.546295 (+ 979us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:09.546319 (+ 24us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:09.553071 (+ 6752us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:09.553115 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:09.554866 (+ 1751us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:09.554898 (+ 32us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:09.555165 (+ 267us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:09.555409 (+ 244us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:09.555427 (+ 18us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:09.572546 (+ 17119us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:09.580784 (+ 8238us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:09.580807 (+ 23us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:09.580823 (+ 16us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:09.581971 (+ 1148us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:09.588920 (+ 6949us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:09.588950 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:09.588970 (+ 20us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:09.595257 (+ 6287us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:09.596741 (+ 1484us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:09.596765 (+ 24us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:09.597014 (+ 249us) client_negotiation.cc:695] Sending connection context 0627 19:32:09.597552 (+ 538us) client_negotiation.cc:225] Negotiation successful 0627 19:32:09.597883 (+ 331us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":19668,"mutex_wait_us":20148,"thread_start_us":18984,"threads_started":1} I0627 19:32:09.616613 24836 heartbeater.cc:346] Connected to a master server at 127.12.131.60:47715 I0627 19:32:09.616955 24836 heartbeater.cc:426] Registering TS with master... I0627 19:32:09.617663 24836 heartbeater.cc:475] Master 127.12.131.60:47715 requested a full tablet report, sending... I0627 19:32:09.620676 24616 ts_manager.cc:98] Registered new tserver with Master: fbd50e06440b4131b656d2627053720a (127.12.131.1:36187) I0627 19:32:09.604176 24829 heartbeater.cc:346] Connected to a master server at 127.12.131.62:54619 I0627 19:32:09.638244 24829 heartbeater.cc:426] Registering TS with master... I0627 19:32:09.638741 24829 heartbeater.cc:475] Master 127.12.131.62:54619 requested a full tablet report, sending... I0627 19:32:09.642391 24474 ts_manager.cc:98] Registered new tserver with Master: fbd50e06440b4131b656d2627053720a (127.12.131.1:36187) I0627 19:32:09.605109 24834 heartbeater.cc:346] Connected to a master server at 127.12.131.61:54937 I0627 19:32:09.646059 24834 heartbeater.cc:426] Registering TS with master... I0627 19:32:09.646538 24834 heartbeater.cc:475] Master 127.12.131.61:54937 requested a full tablet report, sending... I0627 19:32:09.649344 24544 ts_manager.cc:98] Registered new tserver with Master: fbd50e06440b4131b656d2627053720a (127.12.131.1:36187) WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:32:09.825749 24857 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:32:09.828555 24857 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.2 W0627 19:32:09.829032 24857 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:32:09.835026 24682 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:09.835233 24682 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:09.860100 24857 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:32:09.870600 24857 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:32:09.870955 24857 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.2 W0627 19:32:09.871580 24857 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.2 I0627 19:32:09.965618 24857 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.2:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.2.keytab --principal=kudu/127.12.131.2 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/data/info.pb --webserver_interface=127.12.131.2 --webserver_port=0 --tserver_master_addrs=127.12.131.62:54619,127.12.131.61:54937,127.12.131.60:47715 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.2 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:32:10.014283 24857 tablet_server_main.cc:85] Initializing tablet server... I0627 19:32:10.018965 24857 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 234092us Jun 27 19:32:10 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153130, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:32:10.031486 24857 init.cc:339] Logged in from keytab as kudu/127.12.131.2@KRBTEST.COM (short username kudu) I0627 19:32:10.054816 24857 fs_manager.cc:261] Metadata directory not provided I0627 19:32:10.055068 24857 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:32:10.055630 24857 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:32:10.055852 24857 server_base.cc:433] Attempting to create new FS layout instead I0627 19:32:10.080830 24857 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/data/instance: uuid: "de48899c8b404b9e9121c422fa8cdd9a" format_stamp: "Formatted at 2018-06-28 02:32:10 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:10.082005 24857 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/wal/instance: uuid: "de48899c8b404b9e9121c422fa8cdd9a" format_stamp: "Formatted at 2018-06-28 02:32:10 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:10.126709 24857 fs_manager.cc:501] Time spent creating directory manager: real 0.038s user 0.013s sys 0.003s I0627 19:32:10.128726 24857 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:32:10.129354 24857 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:32:10.157809 24857 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.001s sys 0.002s I0627 19:32:10.158094 24857 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/wal uuid: "de48899c8b404b9e9121c422fa8cdd9a" format_stamp: "Formatted at 2018-06-28 02:32:10 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:10.158521 24857 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0627 19:32:10.482280 24663 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:10.482522 24663 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:10.510924 24667 catalog_manager.cc:1064] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:10.511155 24667 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:32:10.687163 24857 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:32:10.701916 24857 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:32:10.706815 24857 tablet_server_main.cc:90] Starting tablet server... W0627 19:32:10.838001 24682 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:10.838237 24682 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:11.485064 24663 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:11.485292 24663 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:11.513705 24667 catalog_manager.cc:1064] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:11.513931 24667 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:32:11.528193 24857 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.2:57672 I0627 19:32:11.529850 24857 webserver.cc:175] Starting webserver on 127.12.131.2:0 I0627 19:32:11.530009 24857 webserver.cc:186] Document root disabled I0627 19:32:11.539539 24857 webserver.cc:313] Webserver started. Bound to: http://127.12.131.2:50847/ I0627 19:32:11.540652 24857 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-1/data/info.pb I0627 19:32:11.541683 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 24857 WARNING: no policy specified for kudu/127.12.131.3@KRBTEST.COM; defaulting to no policy Authenticating as principal test-admin/admin@KRBTEST.COM with password. Principal "kudu/127.12.131.3@KRBTEST.COM" created. I0627 19:32:11.596593 24857 tablet_server_main.cc:93] Tablet server successfully started. Authenticating as principal test-admin/admin@KRBTEST.COM with password. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. Entry for principal kudu/127.12.131.3 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab. I0627 19:32:11.635141 12812 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/data --block_manager=log --rpc_bind_addresses=127.12.131.3:0 --local_ip_for_outbound_sockets=127.12.131.3 --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:54619,127.12.131.61:54937,127.12.131.60:47715 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --superuser_acl=test-admin --user_acl=test-user --rpc_trace_negotiation --rpc_authentication=required --rpc_encryption=required with env {KRB5CCNAME=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/krb5cc,KRB5_CONFIG=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes} W0627 19:32:11.840682 24682 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:11.840893 24682 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry WARNING: Logging before InitGoogleLogging() is written to STDERR W0627 19:32:11.869204 25098 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0627 19:32:11.869722 25098 flags.cc:406] Enabled unsafe flag: --principal=kudu/127.12.131.3 W0627 19:32:11.871062 25098 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0627 19:32:11.908064 25098 flags.cc:406] Enabled experimental flag: --rpc_trace_negotiation=true W0627 19:32:11.910727 25098 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0627 19:32:11.911412 25098 flags.cc:406] Enabled experimental flag: --principal=kudu/127.12.131.3 W0627 19:32:11.912039 25098 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.131.3 I0627 19:32:11.966528 25098 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/wal --rpc_trace_negotiation=true --ipki_server_key_size=1024 --rpc_bind_addresses=127.12.131.3:0 --rpc_server_allow_ephemeral_ports=true --keytab_file=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/krb5kdc/kudu_127.12.131.3.keytab --principal=kudu/127.12.131.3 --rpc_authentication=required --rpc_encryption=required --superuser_acl=test-admin --user_acl=test-user --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/data/info.pb --webserver_interface=127.12.131.3 --webserver_port=0 --tserver_master_addrs=127.12.131.62:54619,127.12.131.61:54937,127.12.131.60:47715 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.12.131.3 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 29c66db48b635b7a1d708c7ddb7e0fcd3c8c33fd build type FASTDEBUG built by jenkins at 27 Jun 2018 18:58:26 PST on quasar-ynjtoc-1.vpc.cloudera.com build id 88045 TSAN enabled I0627 19:32:11.988003 25098 tablet_server_main.cc:85] Initializing tablet server... I0627 19:32:11.993142 25098 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 234592us W0627 19:32:12.487649 24663 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:12.487849 24663 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:12.516393 24667 catalog_manager.cc:1064] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:12.516628 24667 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry Jun 27 19:32:12 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153130, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM Jun 27 19:32:12 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153132, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM I0627 19:32:12.623597 25098 init.cc:339] Logged in from keytab as kudu/127.12.131.3@KRBTEST.COM (short username kudu) I0627 19:32:12.642787 25098 fs_manager.cc:261] Metadata directory not provided I0627 19:32:12.643026 25098 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0627 19:32:12.643455 25098 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0627 19:32:12.643666 25098 server_base.cc:433] Attempting to create new FS layout instead I0627 19:32:12.658571 25098 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/data/instance: uuid: "6c3eb141a6474b4e9051fa11d30c5910" format_stamp: "Formatted at 2018-06-28 02:32:12 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:12.659656 25098 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/wal/instance: uuid: "6c3eb141a6474b4e9051fa11d30c5910" format_stamp: "Formatted at 2018-06-28 02:32:12 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:12.682435 25098 fs_manager.cc:501] Time spent creating directory manager: real 0.022s user 0.015s sys 0.000s I0627 19:32:12.684339 25098 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0627 19:32:12.685027 25098 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0627 19:32:12.709895 25098 fs_manager.cc:417] Time spent opening block manager: real 0.014s user 0.001s sys 0.013s I0627 19:32:12.710139 25098 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/wal uuid: "6c3eb141a6474b4e9051fa11d30c5910" format_stamp: "Formatted at 2018-06-28 02:32:12 on quasar-ynjtoc-1.vpc.cloudera.com" I0627 19:32:12.710561 25098 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0627 19:32:12.843315 24682 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:12.843554 24682 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:32:13.077236 25098 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0627 19:32:13.094434 25098 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0627 19:32:13.109396 25098 tablet_server_main.cc:90] Starting tablet server... W0627 19:32:13.490211 24663 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:13.490442 24663 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry W0627 19:32:13.521507 24667 catalog_manager.cc:1064] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:13.521718 24667 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry Jun 27 19:32:13 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153130, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM Jun 27 19:32:13 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153130, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.2@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:32:13.660795 25089 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:11.579281 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:11.596912 (+ 17631us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:11.597069 (+ 157us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:11.610448 (+ 13379us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:11.612037 (+ 1589us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:11.612068 (+ 31us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:11.613069 (+ 1001us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:11.614244 (+ 1175us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:11.614809 (+ 565us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:11.614904 (+ 95us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:11.618070 (+ 3166us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:11.618095 (+ 25us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:11.619030 (+ 935us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:11.619073 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:11.624572 (+ 5499us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:11.624603 (+ 31us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:11.625164 (+ 561us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:11.625403 (+ 239us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:11.625445 (+ 42us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:13.628669 (+2003224us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:13.651928 (+ 23259us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:13.651958 (+ 30us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:13.651979 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:13.653186 (+ 1207us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:13.655087 (+ 1901us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:13.655112 (+ 25us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:13.655131 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:13.655280 (+ 149us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:13.656978 (+ 1698us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:13.657007 (+ 29us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:13.657473 (+ 466us) client_negotiation.cc:695] Sending connection context 0627 19:32:13.657992 (+ 519us) client_negotiation.cc:225] Negotiation successful 0627 19:32:13.658391 (+ 399us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":16203,"mutex_wait_us":2000102,"thread_start_us":14930,"threads_started":1} I0627 19:32:13.666978 25082 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:11.558017 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:54619 0627 19:32:11.564618 (+ 6601us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:11.564727 (+ 109us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:11.610448 (+ 45721us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:11.619143 (+ 8695us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:11.619169 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:11.619915 (+ 746us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:11.626995 (+ 7080us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:11.627323 (+ 328us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:11.627379 (+ 56us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:11.628154 (+ 775us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:11.628173 (+ 19us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:11.628740 (+ 567us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:11.628772 (+ 32us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:11.630153 (+ 1381us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:11.630181 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:11.630455 (+ 274us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:11.630647 (+ 192us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:11.630659 (+ 12us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:13.635198 (+2004539us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:13.647608 (+ 12410us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:13.647642 (+ 34us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:13.647670 (+ 28us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:13.649152 (+ 1482us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:13.651031 (+ 1879us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:13.651057 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:13.651078 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:13.651313 (+ 235us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:13.663835 (+ 12522us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:13.663863 (+ 28us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:13.664182 (+ 319us) client_negotiation.cc:695] Sending connection context 0627 19:32:13.664556 (+ 374us) client_negotiation.cc:225] Negotiation successful 0627 19:32:13.664885 (+ 329us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":5994,"mutex_wait_us":2008583,"thread_start_us":7212,"threads_started":1} I0627 19:32:13.677181 25088 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:11.579286 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:43192 0627 19:32:11.586292 (+ 7006us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:11.586319 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:11.597202 (+ 10883us) server_negotiation.cc:373] Connection header received 0627 19:32:11.610884 (+ 13682us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:11.610916 (+ 32us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:11.611048 (+ 132us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:11.611190 (+ 142us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:11.615923 (+ 4733us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:11.616314 (+ 391us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:11.622843 (+ 6529us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:11.623884 (+ 1041us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:11.624076 (+ 192us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:13.629528 (+2005452us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:13.629608 (+ 80us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:13.629635 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:13.629726 (+ 91us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:13.643584 (+ 13858us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:13.653826 (+ 10242us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:13.653856 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:13.653875 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:13.654584 (+ 709us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:13.655871 (+ 1287us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:13.655896 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:13.655914 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:13.656378 (+ 464us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:13.667594 (+ 11216us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:13.668025 (+ 431us) server_negotiation.cc:264] Negotiation successful 0627 19:32:13.668388 (+ 363us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":6783,"thread_start_us":6479,"threads_started":1} I0627 19:32:13.679797 25084 heartbeater.cc:346] Connected to a master server at 127.12.131.60:47715 I0627 19:32:13.680030 25083 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:11.559634 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:57553 0627 19:32:11.561980 (+ 2346us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:11.562007 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:11.564899 (+ 2892us) server_negotiation.cc:373] Connection header received 0627 19:32:11.618505 (+ 53606us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:11.618534 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:11.618665 (+ 131us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:11.618812 (+ 147us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:11.627601 (+ 8789us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:11.627982 (+ 381us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:11.628936 (+ 954us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:11.629937 (+ 1001us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:11.630116 (+ 179us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:13.635930 (+2005814us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:13.636008 (+ 78us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:13.636033 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:13.636118 (+ 85us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:13.646969 (+ 10851us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:13.649816 (+ 2847us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:13.649844 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:13.649863 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:13.650504 (+ 641us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:13.662614 (+ 12110us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:13.662643 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:13.662664 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:13.663137 (+ 473us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:13.663492 (+ 355us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:13.677751 (+ 14259us) server_negotiation.cc:264] Negotiation successful 0627 19:32:13.678129 (+ 378us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":2124,"thread_start_us":1893,"threads_started":1} I0627 19:32:13.680121 25084 heartbeater.cc:426] Registering TS with master... I0627 19:32:13.680794 25084 heartbeater.cc:475] Master 127.12.131.60:47715 requested a full tablet report, sending... I0627 19:32:13.684069 24616 ts_manager.cc:98] Registered new tserver with Master: de48899c8b404b9e9121c422fa8cdd9a (127.12.131.2:57672) I0627 19:32:13.686583 25086 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:11.570125 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:54937 0627 19:32:11.576443 (+ 6318us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:11.576650 (+ 207us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:11.612462 (+ 35812us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:11.613121 (+ 659us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:11.613176 (+ 55us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:11.614781 (+ 1605us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:11.615134 (+ 353us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:11.615471 (+ 337us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:11.615508 (+ 37us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:11.616674 (+ 1166us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:11.616700 (+ 26us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:11.617782 (+ 1082us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:11.617826 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:11.619474 (+ 1648us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:11.619504 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:11.619912 (+ 408us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:11.620252 (+ 340us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:11.620273 (+ 21us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:13.661967 (+2041694us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:13.674767 (+ 12800us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:13.674796 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:13.674817 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:13.676064 (+ 1247us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:13.678152 (+ 2088us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:13.678179 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:13.678200 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:13.678356 (+ 156us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:13.682700 (+ 4344us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:13.682725 (+ 25us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:13.683013 (+ 288us) client_negotiation.cc:695] Sending connection context 0627 19:32:13.683364 (+ 351us) client_negotiation.cc:225] Negotiation successful 0627 19:32:13.683704 (+ 340us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":5970,"mutex_wait_us":49,"thread_start_us":5602,"threads_started":1} I0627 19:32:13.687371 25087 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:11.571185 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.2:53575 0627 19:32:11.579194 (+ 8009us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:11.579223 (+ 29us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:11.579265 (+ 42us) server_negotiation.cc:373] Connection header received 0627 19:32:11.612721 (+ 33456us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:11.612752 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:11.612875 (+ 123us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:11.613053 (+ 178us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:11.616064 (+ 3011us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:11.616496 (+ 432us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:11.618002 (+ 1506us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:11.619066 (+ 1064us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:11.619243 (+ 177us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:13.662726 (+2043483us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:13.662808 (+ 82us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:13.662836 (+ 28us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:13.662925 (+ 89us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:13.673173 (+ 10248us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:13.676752 (+ 3579us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:13.676782 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:13.676805 (+ 23us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:13.677469 (+ 664us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:13.679242 (+ 1773us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:13.679270 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:13.679290 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:13.679782 (+ 492us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:13.680117 (+ 335us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:13.683662 (+ 3545us) server_negotiation.cc:264] Negotiation successful 0627 19:32:13.684012 (+ 350us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":7706,"thread_start_us":10105,"threads_started":1} I0627 19:32:13.689632 25081 heartbeater.cc:346] Connected to a master server at 127.12.131.61:54937 I0627 19:32:13.689869 25081 heartbeater.cc:426] Registering TS with master... I0627 19:32:13.690327 25081 heartbeater.cc:475] Master 127.12.131.61:54937 requested a full tablet report, sending... I0627 19:32:13.692734 24544 ts_manager.cc:98] Registered new tserver with Master: de48899c8b404b9e9121c422fa8cdd9a (127.12.131.2:57672) I0627 19:32:13.682260 25076 heartbeater.cc:346] Connected to a master server at 127.12.131.62:54619 I0627 19:32:13.695792 25076 heartbeater.cc:426] Registering TS with master... I0627 19:32:13.696244 25076 heartbeater.cc:475] Master 127.12.131.62:54619 requested a full tablet report, sending... I0627 19:32:13.699297 24474 ts_manager.cc:98] Registered new tserver with Master: de48899c8b404b9e9121c422fa8cdd9a (127.12.131.2:57672) W0627 19:32:13.845973 24682 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0627 19:32:13.846200 24682 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0627 19:32:13.899175 25098 rpc_server.cc:205] RPC server started. Bound to: 127.12.131.3:44527 I0627 19:32:13.900959 25098 webserver.cc:175] Starting webserver on 127.12.131.3:0 I0627 19:32:13.901262 25098 webserver.cc:186] Document root disabled I0627 19:32:13.929111 25098 webserver.cc:313] Webserver started. Bound to: http://127.12.131.3:41781/ I0627 19:32:13.930317 25098 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1530153042096250-12812/minicluster-data/ts-2/data/info.pb I0627 19:32:13.939244 12812 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver as pid 25098 Jun 27 19:32:13 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM I0627 19:32:13.983893 25088 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:13.940654 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:51001 0627 19:32:13.940975 (+ 321us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:13.940996 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:13.942969 (+ 1973us) server_negotiation.cc:373] Connection header received 0627 19:32:13.945561 (+ 2592us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:13.945591 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:13.945730 (+ 139us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:13.945870 (+ 140us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:13.951390 (+ 5520us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:13.951897 (+ 507us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:13.955245 (+ 3348us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:13.956300 (+ 1055us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:13.956493 (+ 193us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:13.968599 (+ 12106us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:13.968634 (+ 35us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:13.968658 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:13.968760 (+ 102us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:13.975750 (+ 6990us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:13.977682 (+ 1932us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:13.977709 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:13.977728 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:13.978331 (+ 603us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:13.978907 (+ 576us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:13.978932 (+ 25us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:13.978951 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:13.979526 (+ 575us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:13.979873 (+ 347us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:13.981635 (+ 1762us) server_negotiation.cc:264] Negotiation successful 0627 19:32:13.982082 (+ 447us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":124} I0627 19:32:13.987538 25098 tablet_server_main.cc:93] Tablet server successfully started. Jun 27 19:32:14 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM Jun 27 19:32:14 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153132, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM Jun 27 19:32:14 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153132, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for kudu/127.12.131.61@KRBTEST.COM Jun 27 19:32:14 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153132, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.131.3@KRBTEST.COM for kudu/127.12.131.60@KRBTEST.COM I0627 19:32:14.043710 25083 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:13.937152 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:34436 0627 19:32:13.937514 (+ 362us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:13.937535 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:13.944371 (+ 6836us) server_negotiation.cc:373] Connection header received 0627 19:32:13.994290 (+ 49919us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:13.994320 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:13.994477 (+ 157us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:13.994619 (+ 142us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.004042 (+ 9423us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.004518 (+ 476us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.005870 (+ 1352us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.006891 (+ 1021us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.007059 (+ 168us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.017522 (+ 10463us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:14.017589 (+ 67us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:14.017614 (+ 25us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:14.017704 (+ 90us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:14.023939 (+ 6235us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.036301 (+ 12362us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.036333 (+ 32us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.036352 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.037083 (+ 731us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.038674 (+ 1591us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.038703 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.038723 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.039185 (+ 462us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:14.039540 (+ 355us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:14.041586 (+ 2046us) server_negotiation.cc:264] Negotiation successful 0627 19:32:14.041934 (+ 348us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":130} I0627 19:32:14.045765 25369 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:13.935086 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.62:54619 0627 19:32:13.944075 (+ 8989us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:13.944180 (+ 105us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:13.993937 (+ 49757us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:13.995199 (+ 1262us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:13.995237 (+ 38us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:13.996112 (+ 875us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:14.003166 (+ 7054us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:14.003543 (+ 377us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:14.003588 (+ 45us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.004771 (+ 1183us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.004800 (+ 29us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:14.005637 (+ 837us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:14.005680 (+ 43us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.007479 (+ 1799us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.007509 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:14.007775 (+ 266us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.009217 (+ 1442us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:14.009239 (+ 22us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:14.016937 (+ 7698us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:14.025477 (+ 8540us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:14.025510 (+ 33us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:14.025549 (+ 39us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:14.029713 (+ 4164us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:14.037664 (+ 7951us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:14.037691 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:14.037712 (+ 21us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:14.038096 (+ 384us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:14.040041 (+ 1945us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:14.040073 (+ 32us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:14.040657 (+ 584us) client_negotiation.cc:695] Sending connection context 0627 19:32:14.041288 (+ 631us) client_negotiation.cc:225] Negotiation successful 0627 19:32:14.041742 (+ 454us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":5935,"mutex_wait_us":3473,"thread_start_us":6226,"threads_started":1} I0627 19:32:14.050519 25367 heartbeater.cc:346] Connected to a master server at 127.12.131.62:54619 I0627 19:32:14.050920 25367 heartbeater.cc:426] Registering TS with master... I0627 19:32:14.051637 25367 heartbeater.cc:475] Master 127.12.131.62:54619 requested a full tablet report, sending... I0627 19:32:14.054780 24474 ts_manager.cc:98] Registered new tserver with Master: 6c3eb141a6474b4e9051fa11d30c5910 (127.12.131.3:44527) I0627 19:32:14.057268 25375 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:13.962849 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:51348 0627 19:32:13.968932 (+ 6083us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:13.968957 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:13.992674 (+ 23717us) server_negotiation.cc:373] Connection header received 0627 19:32:14.002625 (+ 9951us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.002656 (+ 31us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.002785 (+ 129us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.002933 (+ 148us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.013166 (+ 10233us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.013624 (+ 458us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.020239 (+ 6615us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.021370 (+ 1131us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.021577 (+ 207us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.036233 (+ 14656us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:14.036326 (+ 93us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:14.036353 (+ 27us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:14.036467 (+ 114us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:14.042965 (+ 6498us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.048101 (+ 5136us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.048132 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.048151 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.048828 (+ 677us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.051024 (+ 2196us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.051051 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.051069 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.051558 (+ 489us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:14.051881 (+ 323us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:14.054724 (+ 2843us) server_negotiation.cc:264] Negotiation successful 0627 19:32:14.055179 (+ 455us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":5835,"thread_start_us":5449,"threads_started":1} I0627 19:32:14.069684 25373 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:13.948759 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.61:54937 0627 19:32:13.967660 (+ 18901us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:13.967812 (+ 152us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:13.996444 (+ 28632us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:13.997250 (+ 806us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:13.997277 (+ 27us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:13.997956 (+ 679us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:13.998212 (+ 256us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:13.998667 (+ 455us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:13.998711 (+ 44us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:13.999671 (+ 960us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:13.999698 (+ 27us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:14.001313 (+ 1615us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:14.001369 (+ 56us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.003931 (+ 2562us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.004327 (+ 396us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:14.006612 (+ 2285us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.007982 (+ 1370us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:14.009892 (+ 1910us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:14.021055 (+ 11163us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:14.034329 (+ 13274us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:14.034356 (+ 27us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:14.034406 (+ 50us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:14.039751 (+ 5345us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:14.048765 (+ 9014us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:14.048791 (+ 26us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:14.048810 (+ 19us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:14.048980 (+ 170us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:14.060937 (+ 11957us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:14.060966 (+ 29us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:14.061370 (+ 404us) client_negotiation.cc:695] Sending connection context 0627 19:32:14.061969 (+ 599us) client_negotiation.cc:225] Negotiation successful 0627 19:32:14.062325 (+ 356us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":18654,"mutex_wait_us":10398,"thread_start_us":10178,"threads_started":1} I0627 19:32:14.065861 25087 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:13.948622 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.3:36635 0627 19:32:13.948957 (+ 335us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:13.948978 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:13.967932 (+ 18954us) server_negotiation.cc:373] Connection header received 0627 19:32:13.996684 (+ 28752us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:13.996713 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:13.996834 (+ 121us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:13.996961 (+ 127us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:13.998922 (+ 1961us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:13.999352 (+ 430us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.001613 (+ 2261us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.002651 (+ 1038us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.002825 (+ 174us) server_negotiation.cc:581] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.021763 (+ 18938us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:14.024725 (+ 2962us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:14.024758 (+ 33us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:14.024851 (+ 93us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:14.033713 (+ 8862us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.044501 (+ 10788us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.044531 (+ 30us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.044550 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.045209 (+ 659us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.055854 (+ 10645us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.055883 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.055903 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.056433 (+ 530us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:14.056779 (+ 346us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:14.063655 (+ 6876us) server_negotiation.cc:264] Negotiation successful 0627 19:32:14.064041 (+ 386us) negotiation.cc:304] Negotiation complete: OK Metrics: {"mutex_wait_us":2448,"server-negotiator.queue_time_us":102} I0627 19:32:14.073758 25368 heartbeater.cc:346] Connected to a master server at 127.12.131.61:54937 I0627 19:32:14.074015 25368 heartbeater.cc:426] Registering TS with master... I0627 19:32:14.074528 25368 heartbeater.cc:475] Master 127.12.131.61:54937 requested a full tablet report, sending... I0627 19:32:14.076962 24544 ts_manager.cc:98] Registered new tserver with Master: 6c3eb141a6474b4e9051fa11d30c5910 (127.12.131.3:44527) I0627 19:32:14.080399 25376 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:13.962717 (+ 0us) reactor.cc:583] Submitting negotiation task for client connection to 127.12.131.60:47715 0627 19:32:13.992362 (+ 29645us) negotiation.cc:98] Waiting for socket to connect 0627 19:32:13.992557 (+ 195us) client_negotiation.cc:160] Beginning negotiation 0627 19:32:14.002378 (+ 9821us) client_negotiation.cc:237] Sending NEGOTIATE NegotiatePB request 0627 19:32:14.003022 (+ 644us) client_negotiation.cc:254] Received NEGOTIATE NegotiatePB response 0627 19:32:14.003048 (+ 26us) client_negotiation.cc:348] Received NEGOTIATE response from server 0627 19:32:14.003881 (+ 833us) client_negotiation.cc:175] Negotiated authn=SASL 0627 19:32:14.004171 (+ 290us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:14.009337 (+ 5166us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:14.009390 (+ 53us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.013838 (+ 4448us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.013868 (+ 30us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:14.019991 (+ 6123us) client_negotiation.cc:464] Sending TLS_HANDSHAKE message to server 0627 19:32:14.020038 (+ 47us) client_negotiation.cc:237] Sending TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.024550 (+ 4512us) client_negotiation.cc:254] Received TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.024578 (+ 28us) client_negotiation.cc:476] Received TLS_HANDSHAKE response from server 0627 19:32:14.024879 (+ 301us) client_negotiation.cc:502] Negotiated TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.025317 (+ 438us) client_negotiation.cc:552] Initiating SASL GSSAPI handshake 0627 19:32:14.025341 (+ 24us) client_negotiation.cc:577] Calling sasl_client_start() 0627 19:32:14.035521 (+ 10180us) client_negotiation.cc:237] Sending SASL_INITIATE NegotiatePB request 0627 19:32:14.043720 (+ 8199us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:14.043749 (+ 29us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:14.043771 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:14.046333 (+ 2562us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:14.049396 (+ 3063us) client_negotiation.cc:254] Received SASL_CHALLENGE NegotiatePB response 0627 19:32:14.049450 (+ 54us) client_negotiation.cc:622] Received SASL_CHALLENGE response from server 0627 19:32:14.049472 (+ 22us) client_negotiation.cc:687] Calling sasl_client_step() 0627 19:32:14.049634 (+ 162us) client_negotiation.cc:237] Sending SASL_RESPONSE NegotiatePB request 0627 19:32:14.052382 (+ 2748us) client_negotiation.cc:254] Received SASL_SUCCESS NegotiatePB response 0627 19:32:14.052430 (+ 48us) client_negotiation.cc:643] Received SASL_SUCCESS response from server 0627 19:32:14.052738 (+ 308us) client_negotiation.cc:695] Sending connection context 0627 19:32:14.054286 (+ 1548us) client_negotiation.cc:225] Negotiation successful 0627 19:32:14.058142 (+ 3856us) negotiation.cc:304] Negotiation complete: OK Metrics: {"client-negotiator.queue_time_us":29351,"mutex_wait_us":9183,"thread_start_us":23797,"threads_started":1} I0627 19:32:14.082984 25371 heartbeater.cc:346] Connected to a master server at 127.12.131.60:47715 I0627 19:32:14.083174 25371 heartbeater.cc:426] Registering TS with master... I0627 19:32:14.083535 25371 heartbeater.cc:475] Master 127.12.131.60:47715 requested a full tablet report, sending... I0627 19:32:14.083933 25377 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:13.989355 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:45225 0627 19:32:14.001612 (+ 12257us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.001637 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.001675 (+ 38us) server_negotiation.cc:373] Connection header received 0627 19:32:14.001773 (+ 98us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.001796 (+ 23us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.004287 (+ 2491us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.004433 (+ 146us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.006310 (+ 1877us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.006797 (+ 487us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.007946 (+ 1149us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.009743 (+ 1797us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.014121 (+ 4378us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.014495 (+ 374us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:14.014522 (+ 27us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:14.014543 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:14.014619 (+ 76us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:14.037368 (+ 22749us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.046850 (+ 9482us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.046879 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.046898 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.047584 (+ 686us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.056943 (+ 9359us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.056971 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.056989 (+ 18us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.057439 (+ 450us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:14.066779 (+ 9340us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:14.068270 (+ 1491us) server_negotiation.cc:264] Negotiation successful 0627 19:32:14.068779 (+ 509us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":11995,"thread_start_us":11674,"threads_started":1} I0627 19:32:14.086396 24616 ts_manager.cc:98] Registered new tserver with Master: 6c3eb141a6474b4e9051fa11d30c5910 (127.12.131.3:44527) Jun 27 19:32:14 quasar-ynjtoc-1.vpc.cloudera.com krb5kdc[24431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1530153124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.131.62@KRBTEST.COM I0627 19:32:14.113127 25083 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:14.088413 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:36426 0627 19:32:14.088691 (+ 278us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.088714 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.088751 (+ 37us) server_negotiation.cc:373] Connection header received 0627 19:32:14.088855 (+ 104us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.088879 (+ 24us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.088996 (+ 117us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.089132 (+ 136us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.091007 (+ 1875us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.091506 (+ 499us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.093128 (+ 1622us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.094161 (+ 1033us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.094348 (+ 187us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.098445 (+ 4097us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:14.098475 (+ 30us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:14.098499 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:14.098578 (+ 79us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:14.104899 (+ 6321us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.106901 (+ 2002us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.106930 (+ 29us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.106950 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.107604 (+ 654us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.108179 (+ 575us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.108206 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.108225 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.108713 (+ 488us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:14.109086 (+ 373us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:14.110798 (+ 1712us) server_negotiation.cc:264] Negotiation successful 0627 19:32:14.111243 (+ 445us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":77} I0627 19:32:14.118933 12812 external_mini_cluster.cc:490] 3 TS(s) registered with all masters I0627 19:32:14.120270 24484 tablet_service.cc:1112] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000" dest_uuid: "8dbb31411daf4b098e97fab30c7476ba" from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.12.131.62:36426 I0627 19:32:14.120748 24484 raft_consensus.cc:457] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0627 19:32:14.120997 24484 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [term 0 FOLLOWER]: Advancing to term 1 I0627 19:32:14.126026 24484 raft_consensus.cc:479] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } I0627 19:32:14.127444 24484 leader_election.cc:243] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [CANDIDATE]: Term 1 election: Requesting vote from peer 501dc3febd1c4dd191dac8eb0224c0c3 (127.12.131.61:54937) I0627 19:32:14.128914 24484 leader_election.cc:243] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [CANDIDATE]: Term 1 election: Requesting vote from peer aa795690fcf84c9890e0db9a92b11099 (127.12.131.60:47715) I0627 19:32:14.129040 24554 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8dbb31411daf4b098e97fab30c7476ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" I0627 19:32:14.129645 24554 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:32:14.133080 24627 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8dbb31411daf4b098e97fab30c7476ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aa795690fcf84c9890e0db9a92b11099" I0627 19:32:14.133656 24627 raft_consensus.cc:2782] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [term 0 FOLLOWER]: Advancing to term 1 I0627 19:32:14.134562 24554 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dbb31411daf4b098e97fab30c7476ba in term 1. I0627 19:32:14.137868 24455 leader_election.cc:379] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [CANDIDATE]: Term 1 election: Vote granted by peer 501dc3febd1c4dd191dac8eb0224c0c3 (127.12.131.61:54937) I0627 19:32:14.138147 24455 leader_election.cc:270] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0627 19:32:14.138978 24627 raft_consensus.cc:2243] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dbb31411daf4b098e97fab30c7476ba in term 1. I0627 19:32:14.146342 25381 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [term 1 FOLLOWER]: Leader election won for term 1 I0627 19:32:14.147696 24455 leader_election.cc:379] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [CANDIDATE]: Term 1 election: Vote granted by peer aa795690fcf84c9890e0db9a92b11099 (127.12.131.60:47715) I0627 19:32:14.160392 25381 raft_consensus.cc:593] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [term 1 LEADER]: Becoming Leader. State: Replica: 8dbb31411daf4b098e97fab30c7476ba, State: Running, Role: LEADER I0627 19:32:14.161077 25381 consensus_queue.cc:204] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } I0627 19:32:14.176342 25382 sys_catalog.cc:337] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dbb31411daf4b098e97fab30c7476ba. Latest consensus state: current_term: 1 leader_uuid: "8dbb31411daf4b098e97fab30c7476ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:14.177114 25382 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [sys.catalog]: This master's current role is: LEADER I0627 19:32:14.188720 25389 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0627 19:32:14.191581 25389 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0627 19:32:14.216941 25083 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:14.174483 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:36427 0627 19:32:14.175058 (+ 575us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.175085 (+ 27us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.179546 (+ 4461us) server_negotiation.cc:373] Connection header received 0627 19:32:14.179906 (+ 360us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.179934 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.180064 (+ 130us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.180198 (+ 134us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.183180 (+ 2982us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.187705 (+ 4525us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.192106 (+ 4401us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.193191 (+ 1085us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.193353 (+ 162us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.196605 (+ 3252us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:14.196631 (+ 26us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:14.196652 (+ 21us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:14.196731 (+ 79us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:14.203378 (+ 6647us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.205425 (+ 2047us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.205477 (+ 52us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.205501 (+ 24us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.206139 (+ 638us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.206738 (+ 599us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.206769 (+ 31us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.206789 (+ 20us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.207253 (+ 464us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:14.207553 (+ 300us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:14.214691 (+ 7138us) server_negotiation.cc:264] Negotiation successful 0627 19:32:14.215193 (+ 502us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":372} I0627 19:32:14.254102 25375 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:14.188327 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:51006 0627 19:32:14.188652 (+ 325us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.188674 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.210553 (+ 21879us) server_negotiation.cc:373] Connection header received 0627 19:32:14.212558 (+ 2005us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.212583 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.212713 (+ 130us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.212850 (+ 137us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.226533 (+ 13683us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.227056 (+ 523us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.228672 (+ 1616us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.229715 (+ 1043us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.229915 (+ 200us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.232519 (+ 2604us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:14.232550 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:14.232574 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:14.232650 (+ 76us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:14.238679 (+ 6029us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.244512 (+ 5833us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.244539 (+ 27us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.244558 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.245174 (+ 616us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.250490 (+ 5316us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.250516 (+ 26us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.250535 (+ 19us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.250990 (+ 455us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:14.251316 (+ 326us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:14.251993 (+ 677us) server_negotiation.cc:264] Negotiation successful 0627 19:32:14.252445 (+ 452us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":109} I0627 19:32:14.249403 25377 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:14.174325 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:45228 0627 19:32:14.174800 (+ 475us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.174821 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.187172 (+ 12351us) server_negotiation.cc:373] Connection header received 0627 19:32:14.187415 (+ 243us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.187442 (+ 27us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.187573 (+ 131us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.187700 (+ 127us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.201276 (+ 13576us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.201702 (+ 426us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.211971 (+ 10269us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.213067 (+ 1096us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.213246 (+ 179us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.228041 (+ 14795us) server_negotiation.cc:329] Received SASL_INITIATE NegotiatePB request 0627 19:32:14.228072 (+ 31us) server_negotiation.cc:757] Received SASL_INITIATE request from client 0627 19:32:14.228096 (+ 24us) server_negotiation.cc:768] Client requested to use mechanism: GSSAPI 0627 19:32:14.228183 (+ 87us) server_negotiation.cc:797] Calling sasl_server_start() 0627 19:32:14.234650 (+ 6467us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.243341 (+ 8691us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.243400 (+ 59us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.243421 (+ 21us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.244100 (+ 679us) server_negotiation.cc:341] Sending SASL_CHALLENGE NegotiatePB response 0627 19:32:14.244711 (+ 611us) server_negotiation.cc:329] Received SASL_RESPONSE NegotiatePB request 0627 19:32:14.244739 (+ 28us) server_negotiation.cc:831] Received SASL_RESPONSE request from client 0627 19:32:14.244764 (+ 25us) server_negotiation.cc:841] Calling sasl_server_step() 0627 19:32:14.245241 (+ 477us) server_negotiation.cc:341] Sending SASL_SUCCESS NegotiatePB response 0627 19:32:14.246996 (+ 1755us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:14.247200 (+ 204us) server_negotiation.cc:264] Negotiation successful 0627 19:32:14.247657 (+ 457us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":273} I0627 19:32:14.321787 24554 raft_consensus.cc:1081] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [term 1 FOLLOWER]: Refusing update from remote peer 8dbb31411daf4b098e97fab30c7476ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0627 19:32:14.323150 24627 raft_consensus.cc:1081] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [term 1 FOLLOWER]: Refusing update from remote peer 8dbb31411daf4b098e97fab30c7476ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0627 19:32:14.324769 25381 consensus_queue.cc:962] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0627 19:32:14.325884 25381 consensus_queue.cc:962] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0627 19:32:14.370805 25393 sys_catalog.cc:337] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dbb31411daf4b098e97fab30c7476ba. Latest consensus state: current_term: 1 leader_uuid: "8dbb31411daf4b098e97fab30c7476ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:14.371634 25393 sys_catalog.cc:340] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:14.375578 25392 sys_catalog.cc:337] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dbb31411daf4b098e97fab30c7476ba. Latest consensus state: current_term: 1 leader_uuid: "8dbb31411daf4b098e97fab30c7476ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:14.376237 25392 sys_catalog.cc:340] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:14.382403 25382 sys_catalog.cc:337] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8dbb31411daf4b098e97fab30c7476ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:14.383076 25382 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [sys.catalog]: This master's current role is: LEADER I0627 19:32:14.383947 25389 catalog_manager.cc:904] Generated new certificate authority record I0627 19:32:14.386010 25389 catalog_manager.cc:1026] Loading token signing keys... I0627 19:32:14.387280 25392 sys_catalog.cc:337] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8dbb31411daf4b098e97fab30c7476ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:14.389724 25392 sys_catalog.cc:340] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3 [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:14.389945 25381 sys_catalog.cc:337] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8dbb31411daf4b098e97fab30c7476ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:14.390599 25381 sys_catalog.cc:340] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba [sys.catalog]: This master's current role is: LEADER I0627 19:32:14.397188 25393 sys_catalog.cc:337] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8dbb31411daf4b098e97fab30c7476ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dbb31411daf4b098e97fab30c7476ba" member_type: VOTER last_known_addr { host: "127.12.131.62" port: 54619 } } peers { permanent_uuid: "501dc3febd1c4dd191dac8eb0224c0c3" member_type: VOTER last_known_addr { host: "127.12.131.61" port: 54937 } } peers { permanent_uuid: "aa795690fcf84c9890e0db9a92b11099" member_type: VOTER last_known_addr { host: "127.12.131.60" port: 47715 } } } I0627 19:32:14.397912 25393 sys_catalog.cc:340] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099 [sys.catalog]: This master's current role is: FOLLOWER I0627 19:32:14.437067 25389 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 8dbb31411daf4b098e97fab30c7476ba: Generated new TSK 0 I0627 19:32:14.528296 24667 catalog_manager.cc:1062] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: acquiring CA information for follower catalog manager: success I0627 19:32:14.530958 24667 catalog_manager.cc:1090] T 00000000000000000000000000000000 P aa795690fcf84c9890e0db9a92b11099: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 W0627 19:32:14.613000 25377 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:14.601319 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:45231 0627 19:32:14.601628 (+ 309us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.601651 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.606067 (+ 4416us) server_negotiation.cc:373] Connection header received 0627 19:32:14.608460 (+ 2393us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.608486 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.608614 (+ 128us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.608738 (+ 124us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.612173 (+ 3435us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:45231: BlockingRecv error: recv got EOF from 127.12.131.61:45231 (error 108) Metrics: {"server-negotiator.queue_time_us":109} W0627 19:32:14.614588 25083 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:14.600564 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:36430 0627 19:32:14.600969 (+ 405us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.600991 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.609424 (+ 8433us) server_negotiation.cc:373] Connection header received 0627 19:32:14.609665 (+ 241us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.609690 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.609820 (+ 130us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.609943 (+ 123us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.613936 (+ 3993us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.62:36430: BlockingRecv error: recv got EOF from 127.12.131.62:36430 (error 108) Metrics: {"server-negotiator.queue_time_us":209} W0627 19:32:14.631958 25375 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:14.624673 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:51009 0627 19:32:14.628690 (+ 4017us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.628713 (+ 23us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.628754 (+ 41us) server_negotiation.cc:373] Connection header received 0627 19:32:14.628862 (+ 108us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.628887 (+ 25us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.629024 (+ 137us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.629172 (+ 148us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.631132 (+ 1960us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.60:51009: BlockingRecv error: recv got EOF from 127.12.131.60:51009 (error 108) Metrics: {"server-negotiator.queue_time_us":3799} I0627 19:32:14.673447 24474 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.1@KRBTEST.COM'} at 127.12.131.1:36447 W0627 19:32:14.692457 25377 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:14.680577 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:45233 0627 19:32:14.680882 (+ 305us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.680907 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.687130 (+ 6223us) server_negotiation.cc:373] Connection header received 0627 19:32:14.689251 (+ 2121us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.689279 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.689493 (+ 214us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.689633 (+ 140us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.691618 (+ 1985us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:45233: BlockingRecv error: recv got EOF from 127.12.131.61:45233 (error 108) Metrics: {"server-negotiator.queue_time_us":89} I0627 19:32:14.704632 24474 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.2@KRBTEST.COM'} at 127.12.131.2:57553 W0627 19:32:14.753469 25377 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:14.736336 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:45234 0627 19:32:14.736656 (+ 320us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.736678 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.743820 (+ 7142us) server_negotiation.cc:373] Connection header received 0627 19:32:14.744082 (+ 262us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.744110 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.744272 (+ 162us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.744425 (+ 153us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.752615 (+ 8190us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:45234: BlockingRecv error: recv got EOF from 127.12.131.61:45234 (error 108) Metrics: {"server-negotiator.queue_time_us":111} W0627 19:32:14.793853 25377 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:14.783023 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:45235 0627 19:32:14.783369 (+ 346us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.783394 (+ 25us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.790723 (+ 7329us) server_negotiation.cc:373] Connection header received 0627 19:32:14.791003 (+ 280us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.791031 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.791196 (+ 165us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.791333 (+ 137us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.793057 (+ 1724us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:45235: BlockingRecv error: recv got EOF from 127.12.131.61:45235 (error 108) Metrics: {"server-negotiator.queue_time_us":140} I0627 19:32:14.852751 24682 catalog_manager.cc:1062] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: acquiring CA information for follower catalog manager: success W0627 19:32:14.856997 25377 negotiation.cc:313] Failed RPC negotiation. Trace: 0627 19:32:14.843215 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:45236 0627 19:32:14.843580 (+ 365us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.843601 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.844634 (+ 1033us) server_negotiation.cc:373] Connection header received 0627 19:32:14.845028 (+ 394us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.845054 (+ 26us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.845211 (+ 157us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.845351 (+ 140us) server_negotiation.cc:197] Negotiated authn=SASL 0627 19:32:14.856157 (+ 10806us) negotiation.cc:304] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.131.61:45236: BlockingRecv error: recv got EOF from 127.12.131.61:45236 (error 108) Metrics: {"server-negotiator.queue_time_us":158} I0627 19:32:14.858695 24682 catalog_manager.cc:1090] T 00000000000000000000000000000000 P 501dc3febd1c4dd191dac8eb0224c0c3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0 I0627 19:32:14.901367 25377 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:14.875727 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:45237 0627 19:32:14.876063 (+ 336us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.876087 (+ 24us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.881168 (+ 5081us) server_negotiation.cc:373] Connection header received 0627 19:32:14.885294 (+ 4126us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.885324 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.885518 (+ 194us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.885649 (+ 131us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:32:14.890134 (+ 4485us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.890654 (+ 520us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.892365 (+ 1711us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.893461 (+ 1096us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.893633 (+ 172us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.898670 (+ 5037us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:32:14.899127 (+ 457us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:32:14.899257 (+ 130us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:14.899743 (+ 486us) server_negotiation.cc:264] Negotiation successful 0627 19:32:14.899922 (+ 179us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":131} I0627 19:32:14.972316 25375 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:14.961810 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:51015 0627 19:32:14.962130 (+ 320us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:14.962152 (+ 22us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:14.963367 (+ 1215us) server_negotiation.cc:373] Connection header received 0627 19:32:14.963688 (+ 321us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:14.963716 (+ 28us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:14.963876 (+ 160us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:14.964013 (+ 137us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:32:14.964668 (+ 655us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.965085 (+ 417us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.968051 (+ 2966us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:14.969164 (+ 1113us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:14.969366 (+ 202us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:14.970139 (+ 773us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:32:14.970591 (+ 452us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:32:14.970712 (+ 121us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:14.970942 (+ 230us) server_negotiation.cc:264] Negotiation successful 0627 19:32:14.971110 (+ 168us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":127} I0627 19:32:15.064678 24474 master_service.cc:249] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.131.3@KRBTEST.COM'} at 127.12.131.3:34436 I0627 19:32:15.069612 25083 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:15.037047 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.62:36439 0627 19:32:15.039078 (+ 2031us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:15.039107 (+ 29us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:15.050455 (+ 11348us) server_negotiation.cc:373] Connection header received 0627 19:32:15.050770 (+ 315us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:15.050787 (+ 17us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:15.050899 (+ 112us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:15.051008 (+ 109us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:32:15.051799 (+ 791us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:15.053474 (+ 1675us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:15.055374 (+ 1900us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:15.056921 (+ 1547us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:15.057368 (+ 447us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:15.058680 (+ 1312us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:32:15.059056 (+ 376us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:32:15.059153 (+ 97us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:15.059541 (+ 388us) server_negotiation.cc:264] Negotiation successful 0627 19:32:15.059709 (+ 168us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":144} I0627 19:32:15.075707 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 24656 I0627 19:32:15.101740 25375 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:15.068061 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.60:51018 0627 19:32:15.068345 (+ 284us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:15.068366 (+ 21us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:15.073662 (+ 5296us) server_negotiation.cc:373] Connection header received 0627 19:32:15.075340 (+ 1678us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:15.075369 (+ 29us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:15.075956 (+ 587us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:15.076061 (+ 105us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:32:15.086451 (+ 10390us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:15.086896 (+ 445us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:15.097201 (+ 10305us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:15.098299 (+ 1098us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:15.098501 (+ 202us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:15.099621 (+ 1120us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:32:15.099816 (+ 195us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:32:15.099928 (+ 112us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:15.100351 (+ 423us) server_negotiation.cc:264] Negotiation successful 0627 19:32:15.100527 (+ 176us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":79} I0627 19:32:15.104348 25377 negotiation.cc:315] RPC negotiation tracing enabled. Trace: 0627 19:32:15.037066 (+ 0us) reactor.cc:583] Submitting negotiation task for server connection from 127.12.131.61:45240 0627 19:32:15.037402 (+ 336us) server_negotiation.cc:176] Beginning negotiation 0627 19:32:15.037450 (+ 48us) server_negotiation.cc:365] Waiting for connection header 0627 19:32:15.067331 (+ 29881us) server_negotiation.cc:373] Connection header received 0627 19:32:15.076842 (+ 9511us) server_negotiation.cc:329] Received NEGOTIATE NegotiatePB request 0627 19:32:15.076872 (+ 30us) server_negotiation.cc:412] Received NEGOTIATE request from client 0627 19:32:15.077029 (+ 157us) server_negotiation.cc:341] Sending NEGOTIATE NegotiatePB response 0627 19:32:15.077152 (+ 123us) server_negotiation.cc:197] Negotiated authn=TOKEN 0627 19:32:15.078512 (+ 1360us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:15.078941 (+ 429us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:15.095385 (+ 16444us) server_negotiation.cc:329] Received TLS_HANDSHAKE NegotiatePB request 0627 19:32:15.096518 (+ 1133us) server_negotiation.cc:341] Sending TLS_HANDSHAKE NegotiatePB response 0627 19:32:15.096701 (+ 183us) server_negotiation.cc:576] Negotiated auth-only TLSv1 with cipher AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1 0627 19:32:15.102256 (+ 5555us) server_negotiation.cc:329] Received TOKEN_EXCHANGE NegotiatePB request 0627 19:32:15.102484 (+ 228us) server_negotiation.cc:341] Sending TOKEN_EXCHANGE NegotiatePB response 0627 19:32:15.102578 (+ 94us) server_negotiation.cc:901] Waiting for connection context 0627 19:32:15.102997 (+ 419us) server_negotiation.cc:264] Negotiation successful 0627 19:32:15.103159 (+ 162us) negotiation.cc:304] Negotiation complete: OK Metrics: {"server-negotiator.queue_time_us":131} W0627 19:32:15.138911 24456 connection.cc:512] server connection from 127.12.131.1:36447 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:36447): Connection reset by peer (error 104) W0627 19:32:15.139768 24528 connection.cc:512] server connection from 127.12.131.1:35921 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:35921): Connection reset by peer (error 104) W0627 19:32:15.139818 24598 connection.cc:512] server connection from 127.12.131.1:37476 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.1:37476): Connection reset by peer (error 104) I0627 19:32:15.140698 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 24857 W0627 19:32:15.177302 24455 connection.cc:512] server connection from 127.12.131.2:57553 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:57553): Connection reset by peer (error 104) W0627 19:32:15.177302 24526 connection.cc:512] server connection from 127.12.131.2:53575 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:53575): Connection reset by peer (error 104) W0627 19:32:15.177315 24597 connection.cc:512] server connection from 127.12.131.2:43192 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.2:43192): Connection reset by peer (error 104) I0627 19:32:15.178158 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-tserver with pid 25098 W0627 19:32:15.210638 24456 connection.cc:512] server connection from 127.12.131.3:34436 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.3:34436): Connection reset by peer (error 104) W0627 19:32:15.210638 24597 connection.cc:512] server connection from 127.12.131.3:51348 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.3:51348): Connection reset by peer (error 104) W0627 19:32:15.210638 24526 connection.cc:512] server connection from 127.12.131.3:36635 recv error: Network error: failed to read from TLS socket (remote: 127.12.131.3:36635): Connection reset by peer (error 104) I0627 19:32:15.211509 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 24443 I0627 19:32:15.250957 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 24512 I0627 19:32:15.292855 12812 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/tsan/bin/kudu-master with pid 24582 [ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (11765 ms) [----------] 2 tests from ConnectToFollowerMasterTest (23690 ms total) [----------] Global test environment tear-down [==========] 23 tests from 3 test cases ran. (93159 ms total) [ PASSED ] 23 tests. ================== WARNING: ThreadSanitizer: data race (pid=12812) Write of size 8 at 0x7b0800000528 by main thread: #0 operator delete(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (security-itest+0x4eb7a1) #1 std::__1::__libcpp_deallocate(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:236:3 (libkrpc.so+0x8d8ba) #2 std::__1::allocator >::deallocate(std::__1::__tree_node*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1796 (libkrpc.so+0x8d8ba) #3 std::__1::allocator_traits > >::deallocate(std::__1::allocator >&, std::__1::__tree_node*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1555 (libkrpc.so+0x8d8ba) #4 std::__1::__tree, std::__1::allocator >::destroy(std::__1::__tree_node*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1834 (libkrpc.so+0x8d8ba) #5 std::__1::__tree, std::__1::allocator >::~__tree() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1821:3 (libkrpc.so+0x8d856) #6 std::__1::set, std::__1::allocator >::~set() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:400:28 (libkrpc.so+0x8bc79) #7 cxa_at_exit_wrapper(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:386 (security-itest+0x44c2b3) Previous read of size 8 at 0x7b0800000528 by thread T10: #0 std::__1::__tree_end_node*>* std::__1::__tree_next_iter*>*, std::__1::__tree_node_base*>(std::__1::__tree_node_base*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:185:14 (libkrpc.so+0x90356) #1 std::__1::__tree_const_iterator*, long>::operator++() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:921 (libkrpc.so+0x90356) #2 void std::__1::__tree, std::__1::allocator >::__assign_multi*, long> >(std::__1::__tree_const_iterator*, long>, std::__1::__tree_const_iterator*, long>) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1667 (libkrpc.so+0x90356) #3 std::__1::__tree, std::__1::allocator >::operator=(std::__1::__tree, std::__1::allocator > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1575:9 (libkrpc.so+0x901a4) #4 std::__1::set, std::__1::allocator >::operator=(std::__1::set, std::__1::allocator > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:485:21 (libkrpc.so+0x870ba) #5 kudu::rpc::ClientNegotiation::SendNegotiate() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:306 (libkrpc.so+0x870ba) #6 kudu::rpc::ClientNegotiation::Negotiate(std::__1::unique_ptr >*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:171:5 (libkrpc.so+0x8693a) #7 kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime, std::__1::unique_ptr >*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:217:3 (libkrpc.so+0xb02b4) #8 kudu::rpc::Negotiation::RunNegotiation(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:294:9 (libkrpc.so+0xaf42b) #9 kudu::internal::RunnableAdapter const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(scoped_refptr const&, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:356:12 (libkrpc.so+0xc853d) #10 kudu::internal::InvokeHelper const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(kudu::rpc::Connection*, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&)>::MakeItSo(kudu::internal::RunnableAdapter const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, kudu::rpc::Connection*, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:925:14 (libkrpc.so+0xc83df) #11 kudu::internal::Invoker<4, kudu::internal::BindState const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime), void ()(scoped_refptr, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1459:12 (libkrpc.so+0xc8299) #12 kudu::Callback::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0xa342d) #13 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1cb77d) #14 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c74a8) #15 boost::_mfi::mf0::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1d2419) #16 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1d236a) #17 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1d22f3) #18 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1d20e9) #19 boost::function0::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1) #20 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bec04) Thread T10 'client-negotiat' (tid=25446, finished) created by thread T18 at: #0 pthread_create /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (security-itest+0x47dceb) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:556:15 (libkudu_util.so+0x1be62f) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::ThreadPool::* const&)(), kudu::ThreadPool* const&, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1c97c5) #3 kudu::ThreadPool::CreateThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:749:10 (libkudu_util.so+0x1c6ab2) #4 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:556:21 (libkudu_util.so+0x1c527f) #5 kudu::ThreadPool::Submit(std::__1::shared_ptr) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1c6d1f) #6 kudu::ThreadPool::SubmitClosure(kudu::Callback) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:450:10 (libkudu_util.so+0x1c6c61) #7 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:588:3 (libkrpc.so+0xbdf93) #8 kudu::rpc::ReactorThread::FindOrStartConnection(kudu::rpc::ConnectionId const&, kudu::rpc::CredentialsPolicy, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:558:14 (libkrpc.so+0xbe876) #9 kudu::rpc::ReactorThread::AssignOutboundCall(std::__1::shared_ptr) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:342:14 (libkrpc.so+0xbe410) #10 kudu::rpc::AssignOutboundCallTask::Run(kudu::rpc::ReactorThread*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:847:14 (libkrpc.so+0xc5d5b) #11 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:316:10 (libkrpc.so+0xbda51) #12 void ev::base::method_thunk(ev_loop*, ev_async*, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xc6283) #13 ev_invoke_pending /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9780) #14 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:176:3 (libkrpc.so+0xbc855) #15 ev_run /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa88d) #16 ev::loop_ref::run(int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc2748) #17 kudu::rpc::ReactorThread::RunThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:477:9 (libkrpc.so+0xbc99a) #18 boost::_mfi::mf0::operator()(kudu::rpc::ReactorThread*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc69e9) #19 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc693a) #20 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc68c3) #21 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc66b9) #22 boost::function0::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1) #23 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bec04) SUMMARY: ThreadSanitizer: data race /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 in operator delete(void*) ================== ================== WARNING: ThreadSanitizer: data race (pid=12812) Write of size 8 at 0x7b0800000530 by main thread: #0 operator delete(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (security-itest+0x4eb7a1) #1 std::__1::__libcpp_deallocate(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:236:3 (libkrpc.so+0x8d8ba) #2 std::__1::allocator >::deallocate(std::__1::__tree_node*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1796 (libkrpc.so+0x8d8ba) #3 std::__1::allocator_traits > >::deallocate(std::__1::allocator >&, std::__1::__tree_node*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1555 (libkrpc.so+0x8d8ba) #4 std::__1::__tree, std::__1::allocator >::destroy(std::__1::__tree_node*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1834 (libkrpc.so+0x8d8ba) #5 std::__1::__tree, std::__1::allocator >::~__tree() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1821:3 (libkrpc.so+0x8d856) #6 std::__1::set, std::__1::allocator >::~set() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:400:28 (libkrpc.so+0x8bc79) #7 cxa_at_exit_wrapper(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:386 (security-itest+0x44c2b3) Previous read of size 8 at 0x7b0800000530 by thread T10: #0 std::__1::__tree_end_node*>* std::__1::__tree_next_iter*>*, std::__1::__tree_node_base*>(std::__1::__tree_node_base*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:84:24 (libkrpc.so+0x9039c) #1 std::__1::__tree_const_iterator*, long>::operator++() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:921 (libkrpc.so+0x9039c) #2 void std::__1::__tree, std::__1::allocator >::__assign_multi*, long> >(std::__1::__tree_const_iterator*, long>, std::__1::__tree_const_iterator*, long>) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1667 (libkrpc.so+0x9039c) #3 std::__1::__tree, std::__1::allocator >::operator=(std::__1::__tree, std::__1::allocator > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1575:9 (libkrpc.so+0x901a4) #4 std::__1::set, std::__1::allocator >::operator=(std::__1::set, std::__1::allocator > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:485:21 (libkrpc.so+0x870ba) #5 kudu::rpc::ClientNegotiation::SendNegotiate() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:306 (libkrpc.so+0x870ba) #6 kudu::rpc::ClientNegotiation::Negotiate(std::__1::unique_ptr >*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:171:5 (libkrpc.so+0x8693a) #7 kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime, std::__1::unique_ptr >*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:217:3 (libkrpc.so+0xb02b4) #8 kudu::rpc::Negotiation::RunNegotiation(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:294:9 (libkrpc.so+0xaf42b) #9 kudu::internal::RunnableAdapter const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(scoped_refptr const&, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:356:12 (libkrpc.so+0xc853d) #10 kudu::internal::InvokeHelper const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(kudu::rpc::Connection*, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&)>::MakeItSo(kudu::internal::RunnableAdapter const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, kudu::rpc::Connection*, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:925:14 (libkrpc.so+0xc83df) #11 kudu::internal::Invoker<4, kudu::internal::BindState const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime), void ()(scoped_refptr, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1459:12 (libkrpc.so+0xc8299) #12 kudu::Callback::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0xa342d) #13 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1cb77d) #14 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c74a8) #15 boost::_mfi::mf0::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1d2419) #16 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1d236a) #17 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1d22f3) #18 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1d20e9) #19 boost::function0::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1) #20 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bec04) Thread T10 'client-negotiat' (tid=25446, finished) created by thread T18 at: #0 pthread_create /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (security-itest+0x47dceb) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:556:15 (libkudu_util.so+0x1be62f) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::ThreadPool::* const&)(), kudu::ThreadPool* const&, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1c97c5) #3 kudu::ThreadPool::CreateThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:749:10 (libkudu_util.so+0x1c6ab2) #4 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:556:21 (libkudu_util.so+0x1c527f) #5 kudu::ThreadPool::Submit(std::__1::shared_ptr) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1c6d1f) #6 kudu::ThreadPool::SubmitClosure(kudu::Callback) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:450:10 (libkudu_util.so+0x1c6c61) #7 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:588:3 (libkrpc.so+0xbdf93) #8 kudu::rpc::ReactorThread::FindOrStartConnection(kudu::rpc::ConnectionId const&, kudu::rpc::CredentialsPolicy, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:558:14 (libkrpc.so+0xbe876) #9 kudu::rpc::ReactorThread::AssignOutboundCall(std::__1::shared_ptr) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:342:14 (libkrpc.so+0xbe410) #10 kudu::rpc::AssignOutboundCallTask::Run(kudu::rpc::ReactorThread*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:847:14 (libkrpc.so+0xc5d5b) #11 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:316:10 (libkrpc.so+0xbda51) #12 void ev::base::method_thunk(ev_loop*, ev_async*, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xc6283) #13 ev_invoke_pending /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9780) #14 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:176:3 (libkrpc.so+0xbc855) #15 ev_run /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa88d) #16 ev::loop_ref::run(int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc2748) #17 kudu::rpc::ReactorThread::RunThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:477:9 (libkrpc.so+0xbc99a) #18 boost::_mfi::mf0::operator()(kudu::rpc::ReactorThread*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc69e9) #19 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc693a) #20 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc68c3) #21 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc66b9) #22 boost::function0::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1) #23 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bec04) SUMMARY: ThreadSanitizer: data race /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 in operator delete(void*) ================== ================== WARNING: ThreadSanitizer: data race (pid=12812) Write of size 8 at 0x7b0800000538 by main thread: #0 operator delete(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (security-itest+0x4eb7a1) #1 std::__1::__libcpp_deallocate(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:236:3 (libkrpc.so+0x8d8ba) #2 std::__1::allocator >::deallocate(std::__1::__tree_node*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1796 (libkrpc.so+0x8d8ba) #3 std::__1::allocator_traits > >::deallocate(std::__1::allocator >&, std::__1::__tree_node*, unsigned long) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1555 (libkrpc.so+0x8d8ba) #4 std::__1::__tree, std::__1::allocator >::destroy(std::__1::__tree_node*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1834 (libkrpc.so+0x8d8ba) #5 std::__1::__tree, std::__1::allocator >::~__tree() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1821:3 (libkrpc.so+0x8d856) #6 std::__1::set, std::__1::allocator >::~set() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:400:28 (libkrpc.so+0x8bc79) #7 cxa_at_exit_wrapper(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:386 (security-itest+0x44c2b3) Previous read of size 4 at 0x7b080000053c by thread T10: #0 _ZNSt3__19allocatorINS_11__tree_nodeIN4kudu3rpc14RpcFeatureFlagEPvEEE9constructIS4_JRKS4_EEEvPT_DpOT0_ /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1805:35 (libkrpc.so+0x90d18) #1 _ZNSt3__116allocator_traitsINS_9allocatorINS_11__tree_nodeIN4kudu3rpc14RpcFeatureFlagEPvEEEEE11__constructIS5_JRKS5_EEEvNS_17integral_constantIbLb1EEERS8_PT_DpOT0_ /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1715 (libkrpc.so+0x90d18) #2 _ZNSt3__116allocator_traitsINS_9allocatorINS_11__tree_nodeIN4kudu3rpc14RpcFeatureFlagEPvEEEEE9constructIS5_JRKS5_EEEvRS8_PT_DpOT0_ /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1561 (libkrpc.so+0x90d18) #3 _ZNSt3__16__treeIN4kudu3rpc14RpcFeatureFlagENS_4lessIS3_EENS_9allocatorIS3_EEE16__construct_nodeIJRKS3_EEENS_10unique_ptrINS_11__tree_nodeIS3_PvEENS_22__tree_node_destructorINS6_ISF_EEEEEEDpOT_ /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:2192 (libkrpc.so+0x90d18) #4 _ZNSt3__16__treeIN4kudu3rpc14RpcFeatureFlagENS_4lessIS3_EENS_9allocatorIS3_EEE15__emplace_multiIJRKS3_EEENS_15__tree_iteratorIS3_PNS_11__tree_nodeIS3_PvEElEEDpOT_ /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:2240:25 (libkrpc.so+0x90c5a) #5 std::__1::__tree_iterator*, long> std::__1::__tree, std::__1::allocator >::__insert_multi(kudu::rpc::RpcFeatureFlag const&&&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1330:16 (libkrpc.so+0x9034c) #6 void std::__1::__tree, std::__1::allocator >::__assign_multi*, long> >(std::__1::__tree_const_iterator*, long>, std::__1::__tree_const_iterator*, long>) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1668 (libkrpc.so+0x9034c) #7 std::__1::__tree, std::__1::allocator >::operator=(std::__1::__tree, std::__1::allocator > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1575:9 (libkrpc.so+0x901a4) #8 std::__1::set, std::__1::allocator >::operator=(std::__1::set, std::__1::allocator > const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:485:21 (libkrpc.so+0x870ba) #9 kudu::rpc::ClientNegotiation::SendNegotiate() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:306 (libkrpc.so+0x870ba) #10 kudu::rpc::ClientNegotiation::Negotiate(std::__1::unique_ptr >*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:171:5 (libkrpc.so+0x8693a) #11 kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime, std::__1::unique_ptr >*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:217:3 (libkrpc.so+0xb02b4) #12 kudu::rpc::Negotiation::RunNegotiation(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:294:9 (libkrpc.so+0xaf42b) #13 kudu::internal::RunnableAdapter const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(scoped_refptr const&, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:356:12 (libkrpc.so+0xc853d) #14 kudu::internal::InvokeHelper const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(kudu::rpc::Connection*, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&)>::MakeItSo(kudu::internal::RunnableAdapter const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, kudu::rpc::Connection*, kudu::TriStateFlag const&, kudu::TriStateFlag const&, kudu::MonoTime const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:925:14 (libkrpc.so+0xc83df) #15 kudu::internal::Invoker<4, kudu::internal::BindState const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime), void ()(scoped_refptr, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void ()(scoped_refptr const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(kudu::internal::BindStateBase*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/bind_internal.h:1459:12 (libkrpc.so+0xc8299) #16 kudu::Callback::Run() const /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/callback.h:396:12 (libconsensus.so+0xa342d) #17 kudu::ClosureRunnable::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:76:9 (libkudu_util.so+0x1cb77d) #18 kudu::ThreadPool::DispatchThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:686:22 (libkudu_util.so+0x1c74a8) #19 boost::_mfi::mf0::operator()(kudu::ThreadPool*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkudu_util.so+0x1d2419) #20 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkudu_util.so+0x1d236a) #21 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkudu_util.so+0x1d22f3) #22 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkudu_util.so+0x1d20e9) #23 boost::function0::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1) #24 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bec04) Thread T10 'client-negotiat' (tid=25446, finished) created by thread T18 at: #0 pthread_create /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:992 (security-itest+0x47dceb) #1 kudu::Thread::StartThread(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, boost::function const&, unsigned long, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:556:15 (libkudu_util.so+0x1be62f) #2 kudu::Status kudu::Thread::Create(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, void (kudu::ThreadPool::* const&)(), kudu::ThreadPool* const&, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:164:12 (libkudu_util.so+0x1c97c5) #3 kudu::ThreadPool::CreateThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:749:10 (libkudu_util.so+0x1c6ab2) #4 kudu::ThreadPool::DoSubmit(std::__1::shared_ptr, kudu::ThreadPoolToken*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:556:21 (libkudu_util.so+0x1c527f) #5 kudu::ThreadPool::Submit(std::__1::shared_ptr) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:458:10 (libkudu_util.so+0x1c6d1f) #6 kudu::ThreadPool::SubmitClosure(kudu::Callback) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:450:10 (libkudu_util.so+0x1c6c61) #7 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr const&) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:588:3 (libkrpc.so+0xbdf93) #8 kudu::rpc::ReactorThread::FindOrStartConnection(kudu::rpc::ConnectionId const&, kudu::rpc::CredentialsPolicy, scoped_refptr*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:558:14 (libkrpc.so+0xbe876) #9 kudu::rpc::ReactorThread::AssignOutboundCall(std::__1::shared_ptr) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:342:14 (libkrpc.so+0xbe410) #10 kudu::rpc::AssignOutboundCallTask::Run(kudu::rpc::ReactorThread*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:847:14 (libkrpc.so+0xc5d5b) #11 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:316:10 (libkrpc.so+0xbda51) #12 void ev::base::method_thunk(ev_loop*, ev_async*, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0xc6283) #13 ev_invoke_pending /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x9780) #14 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:176:3 (libkrpc.so+0xbc855) #15 ev_run /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xa88d) #16 ev::loop_ref::run(int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0xc2748) #17 kudu::rpc::ReactorThread::RunThread() /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:477:9 (libkrpc.so+0xbc99a) #18 boost::_mfi::mf0::operator()(kudu::rpc::ReactorThread*) const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/mem_fn_template.hpp:49:29 (libkrpc.so+0xc69e9) #19 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:259:9 (libkrpc.so+0xc693a) #20 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/bind/bind.hpp:1222:16 (libkrpc.so+0xc68c3) #21 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:159:11 (libkrpc.so+0xc66b9) #22 boost::function0::operator()() const /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/boost/function/function_template.hpp:770:14 (libkrpc.so+0xb64b1) #23 kudu::Thread::SuperviseThread(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:603:3 (libkudu_util.so+0x1bec04) SUMMARY: ThreadSanitizer: data race /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 in operator delete(void*) ================== ThreadSanitizer: reported 3 warnings